[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:58:18 dist-test-slave-glgz krb5kdc[16108](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:58:18 dist-test-slave-glgz krb5kdc[16108](info): set up 2 sockets
Sep 27 03:58:18 dist-test-slave-glgz krb5kdc[16108](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260927 03:58:19.076038 16101 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Sep 27 03:58:19 dist-test-slave-glgz krb5kdc[16108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481499, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:58:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:58:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.185.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab.
Entry for principal kudu/127.15.185.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab.
I20260927 03:58:19.201663 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:39467
--webserver_interface=127.15.185.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:37365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.185.126:39467
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:58:19.282909 16121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:19.283133 16121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:19.283156 16121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:19.284214 16121 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:58:19.284245 16121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:19.284260 16121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:19.284272 16121 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:58:19.284284 16121 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:58:19.285650 16121 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:37365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.185.126:39467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:39467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:19.285838 16121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:19.286012 16121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:19.290975 16126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:19.291051 16127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:19.291148 16129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:19.291148 16121 server_base.cc:1034] running on GCE node
I20260927 03:58:19.291424 16121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:19.291688 16121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:19.292824 16121 hybrid_clock.cc:648] HybridClock initialized: now 1790481499292809 us; error 26 us; skew 500 ppm
Sep 27 03:58:19 dist-test-slave-glgz krb5kdc[16108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481499, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:19.295332 16121 init.cc:377] Logged in from keytab as kudu/127.15.185.126@KRBTEST.COM (short username kudu)
I20260927 03:58:19.295811 16121 webserver.cc:460] Webserver started at http://127.15.185.126:37255/ using document root <none> and password file <none>
I20260927 03:58:19.296043 16121 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:19.296088 16121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:19.296264 16121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:19.297362 16121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/minicluster-data/master-0/data/instance:
uuid: "63c29503b0a841c4b9a93dcb21e76a72"
format_stamp: "Formatted at 2026-09-27 03:58:19 on dist-test-slave-glgz"
server_key: "4302779386481e4058bd012f65c4a973"
server_key_iv: "044a7329a22742517e4c3e240dc0c6ce"
server_key_version: "encryptionkey@0"
I20260927 03:58:19.297730 16121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/minicluster-data/master-0/wal/instance:
uuid: "63c29503b0a841c4b9a93dcb21e76a72"
format_stamp: "Formatted at 2026-09-27 03:58:19 on dist-test-slave-glgz"
server_key: "4302779386481e4058bd012f65c4a973"
server_key_iv: "044a7329a22742517e4c3e240dc0c6ce"
server_key_version: "encryptionkey@0"
I20260927 03:58:19.299304 16121 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:58:19.300143 16136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:19.300390 16121 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:58:19.300494 16121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/minicluster-data/master-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/minicluster-data/master-0/wal
uuid: "63c29503b0a841c4b9a93dcb21e76a72"
format_stamp: "Formatted at 2026-09-27 03:58:19 on dist-test-slave-glgz"
server_key: "4302779386481e4058bd012f65c4a973"
server_key_iv: "044a7329a22742517e4c3e240dc0c6ce"
server_key_version: "encryptionkey@0"
I20260927 03:58:19.300606 16121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:19.309494 16121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:19.311393 16121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:19.311558 16121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:19.315716 16121 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.126:39467
I20260927 03:58:19.315752 16188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.126:39467 every 8 connection(s)
I20260927 03:58:19.316100 16121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
I20260927 03:58:19.316774 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 16121
I20260927 03:58:19.316885 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/minicluster-data/master-0/wal/instance
I20260927 03:58:19.316982 16189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:19.317549 16101 external_mini_cluster.cc:1428] Setting key 69285db9ac62346a72972b054fee8359
I20260927 03:58:19.319674 16189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63c29503b0a841c4b9a93dcb21e76a72: Bootstrap starting.
I20260927 03:58:19.320238 16189 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63c29503b0a841c4b9a93dcb21e76a72: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:19.320523 16189 log.cc:824] T 00000000000000000000000000000000 P 63c29503b0a841c4b9a93dcb21e76a72: Log is configured to *not* fsync() on all Append() calls
I20260927 03:58:19.321205 16189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63c29503b0a841c4b9a93dcb21e76a72: No bootstrap required, opened a new log
I20260927 03:58:19.322944 16189 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63c29503b0a841c4b9a93dcb21e76a72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63c29503b0a841c4b9a93dcb21e76a72" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 39467 } }
I20260927 03:58:19.323153 16189 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63c29503b0a841c4b9a93dcb21e76a72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:19.323223 16189 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63c29503b0a841c4b9a93dcb21e76a72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63c29503b0a841c4b9a93dcb21e76a72, State: Initialized, Role: FOLLOWER
I20260927 03:58:19.323355 16189 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63c29503b0a841c4b9a93dcb21e76a72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63c29503b0a841c4b9a93dcb21e76a72" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 39467 } }
I20260927 03:58:19.323436 16189 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63c29503b0a841c4b9a93dcb21e76a72 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:58:19.323479 16189 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63c29503b0a841c4b9a93dcb21e76a72 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:58:19.323527 16189 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63c29503b0a841c4b9a93dcb21e76a72 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:19.324031 16189 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63c29503b0a841c4b9a93dcb21e76a72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63c29503b0a841c4b9a93dcb21e76a72" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 39467 } }
I20260927 03:58:19.324156 16189 leader_election.cc:302] T 00000000000000000000000000000000 P 63c29503b0a841c4b9a93dcb21e76a72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63c29503b0a841c4b9a93dcb21e76a72; no voters:
I20260927 03:58:19.324350 16189 leader_election.cc:288] T 00000000000000000000000000000000 P 63c29503b0a841c4b9a93dcb21e76a72 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:58:19.324378 16194 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63c29503b0a841c4b9a93dcb21e76a72 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:58:19.324491 16194 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63c29503b0a841c4b9a93dcb21e76a72 [term 1 LEADER]: Becoming Leader. State: Replica: 63c29503b0a841c4b9a93dcb21e76a72, State: Running, Role: LEADER
I20260927 03:58:19.324570 16194 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63c29503b0a841c4b9a93dcb21e76a72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63c29503b0a841c4b9a93dcb21e76a72" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 39467 } }
I20260927 03:58:19.324616 16189 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63c29503b0a841c4b9a93dcb21e76a72 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:58:19.324973 16196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63c29503b0a841c4b9a93dcb21e76a72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63c29503b0a841c4b9a93dcb21e76a72. Latest consensus state: current_term: 1 leader_uuid: "63c29503b0a841c4b9a93dcb21e76a72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63c29503b0a841c4b9a93dcb21e76a72" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 39467 } } }
I20260927 03:58:19.324999 16195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63c29503b0a841c4b9a93dcb21e76a72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63c29503b0a841c4b9a93dcb21e76a72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63c29503b0a841c4b9a93dcb21e76a72" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 39467 } } }
I20260927 03:58:19.325176 16195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63c29503b0a841c4b9a93dcb21e76a72 [sys.catalog]: This master's current role is: LEADER
I20260927 03:58:19.325155 16196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63c29503b0a841c4b9a93dcb21e76a72 [sys.catalog]: This master's current role is: LEADER
I20260927 03:58:19.325577 16203 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:58:19.326092 16203 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:58:19.327683 16203 catalog_manager.cc:1347] Generated new cluster ID: 850d3e44121547739838b691694a76a8
I20260927 03:58:19.328900 16203 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 03:58:19 dist-test-slave-glgz krb5kdc[16108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481499, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 03:58:19.334532 16191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:19.318413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53574 (local address 127.15.185.126:39467)
0927 03:58:19.318612 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:19.318622 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:19.318826 (+ 204us) server_negotiation.cc:407] Connection header received
0927 03:58:19.321010 (+ 2184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:19.321016 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:19.321061 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:19.321209 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:19.322652 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:19.323850 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:19.325276 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:19.326004 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:19.326127 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:19.330015 (+ 3888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:19.330019 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:19.330019 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:19.330030 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:19.332369 (+ 2339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:19.332887 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:19.332891 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:19.332891 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:19.333009 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:19.333291 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:19.333292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:19.333292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:19.333629 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:19.333815 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:19.334136 (+ 321us) server_negotiation.cc:299] Negotiation successful
0927 03:58:19.334295 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":148,"thread_start_us":81,"threads_started":1}
I20260927 03:58:19.337078 16203 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:58:19.337533 16203 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:58:19.341480 16203 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63c29503b0a841c4b9a93dcb21e76a72: Generated new TSK 0
I20260927 03:58:19.341638 16203 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.185.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab.
Entry for principal kudu/127.15.185.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab.
I20260927 03:58:19.369845 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.65:0
--local_ip_for_outbound_sockets=127.15.185.65
--webserver_interface=127.15.185.65
--webserver_port=0
--tserver_master_addrs=127.15.185.126:39467
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab
--principal=kudu/127.15.185.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:37365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:58:19.451516 16215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:19.451748 16215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:19.451789 16215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:19.452941 16215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:19.453009 16215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:19.453053 16215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.65
I20260927 03:58:19.454528 16215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:37365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab
--principal=kudu/127.15.185.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.185.65
--webserver_port=0
--tserver_master_addrs=127.15.185.126:39467
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.65
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:19.454793 16215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:19.455024 16215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:19.460326 16220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:19.460326 16221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:19.460356 16223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:19.460413 16215 server_base.cc:1034] running on GCE node
I20260927 03:58:19.460769 16215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:19.460958 16215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:19.462092 16215 hybrid_clock.cc:648] HybridClock initialized: now 1790481499462084 us; error 28 us; skew 500 ppm
Sep 27 03:58:19 dist-test-slave-glgz krb5kdc[16108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481499, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:19.464581 16215 init.cc:377] Logged in from keytab as kudu/127.15.185.65@KRBTEST.COM (short username kudu)
I20260927 03:58:19.465016 16215 webserver.cc:460] Webserver started at http://127.15.185.65:45269/ using document root <none> and password file <none>
I20260927 03:58:19.465277 16215 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:19.465322 16215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:19.465487 16215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:19.466558 16215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/minicluster-data/ts-0/data/instance:
uuid: "2652e50d94c34f379b1080b72c929c0a"
format_stamp: "Formatted at 2026-09-27 03:58:19 on dist-test-slave-glgz"
server_key: "ba8327d3c627036dbe4592523a2c773e"
server_key_iv: "2bc936f05e23fc6c9d62d76a2ee9ae4a"
server_key_version: "encryptionkey@0"
I20260927 03:58:19.466907 16215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/minicluster-data/ts-0/wal/instance:
uuid: "2652e50d94c34f379b1080b72c929c0a"
format_stamp: "Formatted at 2026-09-27 03:58:19 on dist-test-slave-glgz"
server_key: "ba8327d3c627036dbe4592523a2c773e"
server_key_iv: "2bc936f05e23fc6c9d62d76a2ee9ae4a"
server_key_version: "encryptionkey@0"
I20260927 03:58:19.468212 16215 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:58:19.468938 16230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:19.469101 16215 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:19.469184 16215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/minicluster-data/ts-0/wal
uuid: "2652e50d94c34f379b1080b72c929c0a"
format_stamp: "Formatted at 2026-09-27 03:58:19 on dist-test-slave-glgz"
server_key: "ba8327d3c627036dbe4592523a2c773e"
server_key_iv: "2bc936f05e23fc6c9d62d76a2ee9ae4a"
server_key_version: "encryptionkey@0"
I20260927 03:58:19.469235 16215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:19.484695 16215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:19.486646 16215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:19.486758 16215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:19.486939 16215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:58:19.487339 16215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:58:19.487388 16215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:19.487421 16215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:58:19.487437 16215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:19.493060 16215 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.65:35601
I20260927 03:58:19.493139 16343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.65:35601 every 8 connection(s)
I20260927 03:58:19.493455 16215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
I20260927 03:58:19.494875 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 16215
I20260927 03:58:19.494971 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/minicluster-data/ts-0/wal/instance
I20260927 03:58:19.495154 16101 external_mini_cluster.cc:1428] Setting key 90a90df9ec0d2947946fb87810065d14
Sep 27 03:58:19 dist-test-slave-glgz krb5kdc[16108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481499, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.65@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.185.66@KRBTEST.COM; defaulting to no policy
I20260927 03:58:19.503665 16191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:19.494220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:44991 (local address 127.15.185.126:39467)
0927 03:58:19.494267 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:19.494270 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:19.494486 (+ 216us) server_negotiation.cc:407] Connection header received
0927 03:58:19.495151 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:19.495152 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:19.495163 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:19.495240 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:19.496583 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:19.497055 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:19.498118 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:19.498344 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:19.498430 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:19.500565 (+ 2135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:19.500567 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:19.500567 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:19.500576 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:19.502194 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:19.502593 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:19.502594 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:19.502596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:19.502673 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:19.502914 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:19.502915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:19.502916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:19.503105 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:19.503177 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:19.503479 (+ 302us) server_negotiation.cc:299] Negotiation successful
0927 03:58:19.503566 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 03:58:19.503882 16346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:19.494210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:39467 (local address 127.15.185.65:44991)
0927 03:58:19.494365 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:19.494402 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:19.495012 (+ 610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:19.495362 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:19.495366 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:19.496215 (+ 849us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:19.496499 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:19.496505 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:19.497155 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:19.497156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:19.497950 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:19.497952 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:19.498439 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:19.498440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:19.498583 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:19.499128 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:19.499136 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:19.500470 (+ 1334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:19.502277 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:19.502282 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:19.502284 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:19.502497 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:19.502746 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:19.502747 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:19.502747 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:19.502839 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:19.503174 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:19.503181 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:19.503311 (+ 130us) client_negotiation.cc:769] Sending connection context
0927 03:58:19.503454 (+ 143us) client_negotiation.cc:240] Negotiation successful
0927 03:58:19.503644 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":78,"threads_started":1}
I20260927 03:58:19.504472 16344 heartbeater.cc:347] Connected to a master server at 127.15.185.126:39467
I20260927 03:58:19.504595 16344 heartbeater.cc:464] Registering TS with master...
I20260927 03:58:19.504776 16344 heartbeater.cc:511] Master 127.15.185.126:39467 requested a full tablet report, sending...
I20260927 03:58:19.505280 16153 ts_manager.cc:194] Registered new tserver with Master: 2652e50d94c34f379b1080b72c929c0a (127.15.185.65:35601)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.66@KRBTEST.COM" created.
I20260927 03:58:19.505941 16153 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.185.65@KRBTEST.COM'} at 127.15.185.65:44991
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab.
Entry for principal kudu/127.15.185.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab.
I20260927 03:58:19.521917 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.66:0
--local_ip_for_outbound_sockets=127.15.185.66
--webserver_interface=127.15.185.66
--webserver_port=0
--tserver_master_addrs=127.15.185.126:39467
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab
--principal=kudu/127.15.185.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:37365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:58:19.605792 16349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:19.605996 16349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:19.606019 16349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:19.607257 16349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:19.607297 16349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:19.607323 16349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.66
I20260927 03:58:19.608768 16349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:37365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab
--principal=kudu/127.15.185.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.185.66
--webserver_port=0
--tserver_master_addrs=127.15.185.126:39467
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.66
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:19.609025 16349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:19.609251 16349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 03:58:19.614233 16349 server_base.cc:1034] running on GCE node
W20260927 03:58:19.614259 16357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:19.614300 16355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:19.614513 16354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:19.614706 16349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:19.614902 16349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:19.616040 16349 hybrid_clock.cc:648] HybridClock initialized: now 1790481499616029 us; error 27 us; skew 500 ppm
Sep 27 03:58:19 dist-test-slave-glgz krb5kdc[16108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481499, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:19.618492 16349 init.cc:377] Logged in from keytab as kudu/127.15.185.66@KRBTEST.COM (short username kudu)
I20260927 03:58:19.618916 16349 webserver.cc:460] Webserver started at http://127.15.185.66:45945/ using document root <none> and password file <none>
I20260927 03:58:19.619186 16349 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:19.619246 16349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:19.619402 16349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:19.620434 16349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/minicluster-data/ts-1/data/instance:
uuid: "528ecd7334154cd79a8e65198151d560"
format_stamp: "Formatted at 2026-09-27 03:58:19 on dist-test-slave-glgz"
server_key: "8f55727949724e34ebbde5e9f7c22acb"
server_key_iv: "86281e01f7ecf13bb55103e44ac7bad5"
server_key_version: "encryptionkey@0"
I20260927 03:58:19.620779 16349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/minicluster-data/ts-1/wal/instance:
uuid: "528ecd7334154cd79a8e65198151d560"
format_stamp: "Formatted at 2026-09-27 03:58:19 on dist-test-slave-glgz"
server_key: "8f55727949724e34ebbde5e9f7c22acb"
server_key_iv: "86281e01f7ecf13bb55103e44ac7bad5"
server_key_version: "encryptionkey@0"
I20260927 03:58:19.622031 16349 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:58:19.622844 16364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:19.623108 16349 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:58:19.623214 16349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/minicluster-data/ts-1/wal
uuid: "528ecd7334154cd79a8e65198151d560"
format_stamp: "Formatted at 2026-09-27 03:58:19 on dist-test-slave-glgz"
server_key: "8f55727949724e34ebbde5e9f7c22acb"
server_key_iv: "86281e01f7ecf13bb55103e44ac7bad5"
server_key_version: "encryptionkey@0"
I20260927 03:58:19.623299 16349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:19.644295 16349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:19.646546 16349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:19.646729 16349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:19.646979 16349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:58:19.647327 16349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:58:19.647385 16349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:19.647420 16349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:58:19.647449 16349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:19.653273 16349 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.66:43105
I20260927 03:58:19.653308 16477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.66:43105 every 8 connection(s)
I20260927 03:58:19.653637 16349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
I20260927 03:58:19.657167 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 16349
I20260927 03:58:19.657248 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/minicluster-data/ts-1/wal/instance
I20260927 03:58:19.657408 16101 external_mini_cluster.cc:1428] Setting key a57f58536358641ec197cfc3dde800e1
Sep 27 03:58:19 dist-test-slave-glgz krb5kdc[16108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481499, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.66@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 03:58:19.663029 16191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:19.654373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:59459 (local address 127.15.185.126:39467)
0927 03:58:19.654422 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:19.654424 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:19.654629 (+ 205us) server_negotiation.cc:407] Connection header received
0927 03:58:19.655130 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:19.655131 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:19.655140 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:19.655200 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:19.656107 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:19.656589 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:19.657788 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:19.657990 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:19.658075 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:19.660246 (+ 2171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:19.660248 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:19.660248 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:19.660255 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:19.661698 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:19.662105 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:19.662107 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:19.662107 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:19.662164 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:19.662401 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:19.662401 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:19.662402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:19.662513 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:19.662590 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:19.662859 (+ 269us) server_negotiation.cc:299] Negotiation successful
0927 03:58:19.662938 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 03:58:19.663255 16480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:19.654337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:39467 (local address 127.15.185.66:59459)
0927 03:58:19.654492 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:19.654533 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:19.655015 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:19.655257 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:19.655260 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:19.655825 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:19.656026 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:19.656030 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:19.656669 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:19.656670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:19.657698 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:19.657701 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:19.658084 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:19.658085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:19.658210 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:19.658718 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:19.658726 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:19.660151 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:19.661796 (+ 1645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:19.661803 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:19.661805 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:19.662021 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:19.662232 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:19.662233 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:19.662233 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:19.662334 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:19.662579 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:19.662594 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:19.662732 (+ 138us) client_negotiation.cc:769] Sending connection context
0927 03:58:19.662839 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 03:58:19.662990 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":98,"threads_started":1}
I20260927 03:58:19.663638 16478 heartbeater.cc:347] Connected to a master server at 127.15.185.126:39467
I20260927 03:58:19.663738 16478 heartbeater.cc:464] Registering TS with master...
I20260927 03:58:19.663910 16478 heartbeater.cc:511] Master 127.15.185.126:39467 requested a full tablet report, sending...
I20260927 03:58:19.664279 16153 ts_manager.cc:194] Registered new tserver with Master: 528ecd7334154cd79a8e65198151d560 (127.15.185.66:43105)
I20260927 03:58:19.664662 16153 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.185.66@KRBTEST.COM'} at 127.15.185.66:59459
WARNING: no policy specified for kudu/127.15.185.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab.
Entry for principal kudu/127.15.185.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab.
I20260927 03:58:19.682490 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.67:0
--local_ip_for_outbound_sockets=127.15.185.67
--webserver_interface=127.15.185.67
--webserver_port=0
--tserver_master_addrs=127.15.185.126:39467
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab
--principal=kudu/127.15.185.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:37365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:58:19.764324 16483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:19.764567 16483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:19.764606 16483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:19.765733 16483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:19.765801 16483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:19.765846 16483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.67
I20260927 03:58:19.767329 16483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:37365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab
--principal=kudu/127.15.185.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.185.67
--webserver_port=0
--tserver_master_addrs=127.15.185.126:39467
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.67
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:19.767596 16483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:19.767818 16483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:19.772699 16488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:19.772751 16491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:19.772691 16489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:19.772838 16483 server_base.cc:1034] running on GCE node
I20260927 03:58:19.773144 16483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:19.773378 16483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:19.774552 16483 hybrid_clock.cc:648] HybridClock initialized: now 1790481499774524 us; error 34 us; skew 500 ppm
Sep 27 03:58:19 dist-test-slave-glgz krb5kdc[16108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481499, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:19.777127 16483 init.cc:377] Logged in from keytab as kudu/127.15.185.67@KRBTEST.COM (short username kudu)
I20260927 03:58:19.777595 16483 webserver.cc:460] Webserver started at http://127.15.185.67:39621/ using document root <none> and password file <none>
I20260927 03:58:19.777827 16483 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:19.777894 16483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:19.778047 16483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:19.779166 16483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/minicluster-data/ts-2/data/instance:
uuid: "96efb17efefe464089dc93a793095d78"
format_stamp: "Formatted at 2026-09-27 03:58:19 on dist-test-slave-glgz"
server_key: "9615ee74ab2f9ffbc4bf557865c0ee5b"
server_key_iv: "db2fbb82daad70563f777891f257be8c"
server_key_version: "encryptionkey@0"
I20260927 03:58:19.779525 16483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/minicluster-data/ts-2/wal/instance:
uuid: "96efb17efefe464089dc93a793095d78"
format_stamp: "Formatted at 2026-09-27 03:58:19 on dist-test-slave-glgz"
server_key: "9615ee74ab2f9ffbc4bf557865c0ee5b"
server_key_iv: "db2fbb82daad70563f777891f257be8c"
server_key_version: "encryptionkey@0"
I20260927 03:58:19.780772 16483 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:58:19.781473 16498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:19.781643 16483 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:58:19.781744 16483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/minicluster-data/ts-2/wal
uuid: "96efb17efefe464089dc93a793095d78"
format_stamp: "Formatted at 2026-09-27 03:58:19 on dist-test-slave-glgz"
server_key: "9615ee74ab2f9ffbc4bf557865c0ee5b"
server_key_iv: "db2fbb82daad70563f777891f257be8c"
server_key_version: "encryptionkey@0"
I20260927 03:58:19.781813 16483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:19.796617 16483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:19.798944 16483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:19.799137 16483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:19.799366 16483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:58:19.799677 16483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:58:19.799731 16483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:19.799772 16483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:58:19.799800 16483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:19.805867 16483 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.67:45217
I20260927 03:58:19.805908 16611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.67:45217 every 8 connection(s)
I20260927 03:58:19.806226 16483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
I20260927 03:58:19.807142 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 16483
I20260927 03:58:19.807255 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481498018407-16101-0/minicluster-data/ts-2/wal/instance
I20260927 03:58:19.807428 16101 external_mini_cluster.cc:1428] Setting key bc3fc45e8105b5d1ee957f524feac471
Sep 27 03:58:19 dist-test-slave-glgz krb5kdc[16108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481499, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.67@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 03:58:19.815485 16191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:19.806870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:51709 (local address 127.15.185.126:39467)
0927 03:58:19.806945 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:19.806948 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:19.807181 (+ 233us) server_negotiation.cc:407] Connection header received
0927 03:58:19.807825 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:19.807826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:19.807837 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:19.807884 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:19.809023 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:19.809501 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:19.810443 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:19.810622 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:19.810670 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:19.812805 (+ 2135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:19.812807 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:19.812807 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:19.812815 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:19.814039 (+ 1224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:19.814466 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:19.814467 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:19.814467 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:19.814527 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:19.814796 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:19.814797 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:19.814798 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:19.814915 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:19.814990 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:19.815311 (+ 321us) server_negotiation.cc:299] Negotiation successful
0927 03:58:19.815388 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 03:58:19.815685 16614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:19.806833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:39467 (local address 127.15.185.67:51709)
0927 03:58:19.807009 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:19.807087 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:19.807738 (+ 651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:19.807960 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:19.807964 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:19.808750 (+ 786us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:19.808944 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:19.808949 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:19.809577 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:19.809578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:19.810374 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:19.810375 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:19.810697 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:19.810698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:19.810816 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:19.811375 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:19.811384 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:19.812707 (+ 1323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:19.814144 (+ 1437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:19.814151 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:19.814153 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:19.814378 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:19.814616 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:19.814617 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:19.814618 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:19.814727 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:19.814982 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:19.814988 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:19.815164 (+ 176us) client_negotiation.cc:769] Sending connection context
0927 03:58:19.815290 (+ 126us) client_negotiation.cc:240] Negotiation successful
0927 03:58:19.815456 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":96,"threads_started":1}
I20260927 03:58:19.816113 16612 heartbeater.cc:347] Connected to a master server at 127.15.185.126:39467
I20260927 03:58:19.816203 16612 heartbeater.cc:464] Registering TS with master...
I20260927 03:58:19.816346 16612 heartbeater.cc:511] Master 127.15.185.126:39467 requested a full tablet report, sending...
I20260927 03:58:19.816681 16153 ts_manager.cc:194] Registered new tserver with Master: 96efb17efefe464089dc93a793095d78 (127.15.185.67:45217)
I20260927 03:58:19.817054 16153 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.185.67@KRBTEST.COM'} at 127.15.185.67:51709
I20260927 03:58:19.819510 16101 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 03:58:19 dist-test-slave-glgz krb5kdc[16108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481499, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 03:58:19 dist-test-slave-glgz krb5kdc[16108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481499, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.185.65@KRBTEST.COM
I20260927 03:58:19.842789 16619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:19.834986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58418 (local address 127.15.185.65:35601)
0927 03:58:19.835177 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:19.835182 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:19.835220 (+ 38us) server_negotiation.cc:407] Connection header received
0927 03:58:19.835250 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:19.835253 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:19.835269 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:19.835360 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:19.836010 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:19.836725 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:19.837403 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:19.837707 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:19.837761 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:19.839856 (+ 2095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:19.839864 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:19.839864 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:19.839870 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:19.841590 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:19.841930 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:19.841934 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:19.841935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:19.842001 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:19.842219 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:19.842220 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:19.842221 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:19.842364 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:19.842512 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:19.842590 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 03:58:19.842670 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":89,"threads_started":1}
W20260927 03:58:19.843200 16258 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58418
Sep 27 03:58:19 dist-test-slave-glgz krb5kdc[16108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481499, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 03:58:19 dist-test-slave-glgz krb5kdc[16108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481499, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.185.65@KRBTEST.COM
I20260927 03:58:19.864353 16619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:19.857393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58424 (local address 127.15.185.65:35601)
0927 03:58:19.857460 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:19.857462 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:19.857548 (+ 86us) server_negotiation.cc:407] Connection header received
0927 03:58:19.857661 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:19.857663 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:19.857683 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:19.857746 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:19.858307 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:19.858792 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:19.859611 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:19.859818 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:19.859903 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:19.862023 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:19.862025 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:19.862025 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:19.862032 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:19.863248 (+ 1216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:19.863574 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:19.863575 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:19.863575 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:19.863646 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:19.863845 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:19.863846 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:19.863846 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:19.863949 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:19.864069 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:19.864167 (+ 98us) server_negotiation.cc:299] Negotiation successful
0927 03:58:19.864249 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 03:58:19.864992 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 16215
I20260927 03:58:19.869550 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 16349
I20260927 03:58:19.873495 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 16483
I20260927 03:58:19.877586 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 16121
2026-09-27T03:58:19Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1866 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:58:19 dist-test-slave-glgz krb5kdc[16626](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:58:19 dist-test-slave-glgz krb5kdc[16626](info): set up 2 sockets
Sep 27 03:58:19 dist-test-slave-glgz krb5kdc[16626](info): commencing operation
krb5kdc: starting...
W20260927 03:58:20.906790 16101 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Sep 27 03:58:20 dist-test-slave-glgz krb5kdc[16626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:58:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:58:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.185.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab.
Entry for principal kudu/127.15.185.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab.
I20260927 03:58:21.025928 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:36197
--webserver_interface=127.15.185.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:44859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.185.126:36197
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:58:21.106419 16639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:21.106644 16639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:21.106666 16639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:21.107913 16639 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:58:21.107960 16639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:21.107975 16639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:21.107988 16639 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:58:21.108000 16639 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:58:21.109510 16639 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:44859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.185.126:36197
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:36197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:21.109786 16639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:21.110031 16639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:21.115069 16644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:21.115073 16647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:21.115078 16645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:21.115240 16639 server_base.cc:1034] running on GCE node
I20260927 03:58:21.115561 16639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:21.115862 16639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:21.117023 16639 hybrid_clock.cc:648] HybridClock initialized: now 1790481501117001 us; error 43 us; skew 500 ppm
Sep 27 03:58:21 dist-test-slave-glgz krb5kdc[16626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481501, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:21.119529 16639 init.cc:377] Logged in from keytab as kudu/127.15.185.126@KRBTEST.COM (short username kudu)
I20260927 03:58:21.120015 16639 webserver.cc:460] Webserver started at http://127.15.185.126:33597/ using document root <none> and password file <none>
I20260927 03:58:21.120268 16639 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:21.120316 16639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:21.120469 16639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:21.121565 16639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/minicluster-data/master-0/data/instance:
uuid: "766ebb4b884049e9802ba427195bb76b"
format_stamp: "Formatted at 2026-09-27 03:58:21 on dist-test-slave-glgz"
server_key: "a85bc8cb3d059f14754d42bf8aa75de7"
server_key_iv: "47c341de05e72197cf1eb9634410adfa"
server_key_version: "encryptionkey@0"
I20260927 03:58:21.122031 16639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/minicluster-data/master-0/wal/instance:
uuid: "766ebb4b884049e9802ba427195bb76b"
format_stamp: "Formatted at 2026-09-27 03:58:21 on dist-test-slave-glgz"
server_key: "a85bc8cb3d059f14754d42bf8aa75de7"
server_key_iv: "47c341de05e72197cf1eb9634410adfa"
server_key_version: "encryptionkey@0"
I20260927 03:58:21.124152 16639 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.001s
I20260927 03:58:21.125197 16654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:21.125399 16639 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:58:21.125491 16639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/minicluster-data/master-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/minicluster-data/master-0/wal
uuid: "766ebb4b884049e9802ba427195bb76b"
format_stamp: "Formatted at 2026-09-27 03:58:21 on dist-test-slave-glgz"
server_key: "a85bc8cb3d059f14754d42bf8aa75de7"
server_key_iv: "47c341de05e72197cf1eb9634410adfa"
server_key_version: "encryptionkey@0"
I20260927 03:58:21.125599 16639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:21.136354 16639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:21.138377 16639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:21.138553 16639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:21.142496 16639 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.126:36197
I20260927 03:58:21.142578 16706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.126:36197 every 8 connection(s)
I20260927 03:58:21.142947 16639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
I20260927 03:58:21.143915 16707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:21.146219 16707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 766ebb4b884049e9802ba427195bb76b: Bootstrap starting.
I20260927 03:58:21.146831 16707 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 766ebb4b884049e9802ba427195bb76b: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:21.147187 16707 log.cc:824] T 00000000000000000000000000000000 P 766ebb4b884049e9802ba427195bb76b: Log is configured to *not* fsync() on all Append() calls
I20260927 03:58:21.147910 16707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 766ebb4b884049e9802ba427195bb76b: No bootstrap required, opened a new log
I20260927 03:58:21.149461 16707 raft_consensus.cc:348] T 00000000000000000000000000000000 P 766ebb4b884049e9802ba427195bb76b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "766ebb4b884049e9802ba427195bb76b" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 36197 } }
I20260927 03:58:21.149612 16707 raft_consensus.cc:374] T 00000000000000000000000000000000 P 766ebb4b884049e9802ba427195bb76b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:21.149637 16707 raft_consensus.cc:729] T 00000000000000000000000000000000 P 766ebb4b884049e9802ba427195bb76b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 766ebb4b884049e9802ba427195bb76b, State: Initialized, Role: FOLLOWER
I20260927 03:58:21.149807 16707 consensus_queue.cc:260] T 00000000000000000000000000000000 P 766ebb4b884049e9802ba427195bb76b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "766ebb4b884049e9802ba427195bb76b" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 36197 } }
I20260927 03:58:21.149870 16707 raft_consensus.cc:388] T 00000000000000000000000000000000 P 766ebb4b884049e9802ba427195bb76b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:58:21.149891 16707 raft_consensus.cc:482] T 00000000000000000000000000000000 P 766ebb4b884049e9802ba427195bb76b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:58:21.149955 16707 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 766ebb4b884049e9802ba427195bb76b [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:21.150461 16707 raft_consensus.cc:504] T 00000000000000000000000000000000 P 766ebb4b884049e9802ba427195bb76b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "766ebb4b884049e9802ba427195bb76b" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 36197 } }
I20260927 03:58:21.150599 16707 leader_election.cc:302] T 00000000000000000000000000000000 P 766ebb4b884049e9802ba427195bb76b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 766ebb4b884049e9802ba427195bb76b; no voters:
I20260927 03:58:21.150663 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 16639
I20260927 03:58:21.150748 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/minicluster-data/master-0/wal/instance
I20260927 03:58:21.150785 16707 leader_election.cc:288] T 00000000000000000000000000000000 P 766ebb4b884049e9802ba427195bb76b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:58:21.150818 16710 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 766ebb4b884049e9802ba427195bb76b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:58:21.150928 16101 external_mini_cluster.cc:1428] Setting key 8271e2e1172fb53e5f676895a08d77cd
I20260927 03:58:21.150980 16710 raft_consensus.cc:686] T 00000000000000000000000000000000 P 766ebb4b884049e9802ba427195bb76b [term 1 LEADER]: Becoming Leader. State: Replica: 766ebb4b884049e9802ba427195bb76b, State: Running, Role: LEADER
I20260927 03:58:21.151034 16707 sys_catalog.cc:564] T 00000000000000000000000000000000 P 766ebb4b884049e9802ba427195bb76b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:58:21.151098 16710 consensus_queue.cc:237] T 00000000000000000000000000000000 P 766ebb4b884049e9802ba427195bb76b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "766ebb4b884049e9802ba427195bb76b" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 36197 } }
I20260927 03:58:21.151425 16710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 766ebb4b884049e9802ba427195bb76b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 766ebb4b884049e9802ba427195bb76b. Latest consensus state: current_term: 1 leader_uuid: "766ebb4b884049e9802ba427195bb76b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "766ebb4b884049e9802ba427195bb76b" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 36197 } } }
I20260927 03:58:21.151537 16710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 766ebb4b884049e9802ba427195bb76b [sys.catalog]: This master's current role is: LEADER
I20260927 03:58:21.151847 16711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 766ebb4b884049e9802ba427195bb76b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "766ebb4b884049e9802ba427195bb76b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "766ebb4b884049e9802ba427195bb76b" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 36197 } } }
I20260927 03:58:21.151937 16711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 766ebb4b884049e9802ba427195bb76b [sys.catalog]: This master's current role is: LEADER
I20260927 03:58:21.152431 16718 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:58:21.152952 16718 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:58:21.154533 16718 catalog_manager.cc:1347] Generated new cluster ID: 442f7821c2584b2ab80581d00e08b374
I20260927 03:58:21.154584 16718 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 03:58:21 dist-test-slave-glgz krb5kdc[16626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 03:58:21.163030 16717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:21.151478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56510 (local address 127.15.185.126:36197)
0927 03:58:21.152208 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:21.152219 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:21.152260 (+ 41us) server_negotiation.cc:407] Connection header received
0927 03:58:21.152391 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:21.152399 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:21.152495 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:21.153373 (+ 878us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:21.153452 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:21.154849 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:21.155763 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:21.156222 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:21.156363 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:21.158894 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:21.158898 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:21.158898 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:21.158911 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:21.161356 (+ 2445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:21.161771 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:21.161774 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:21.161774 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:21.161883 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:21.162118 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:21.162119 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:21.162120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:21.162375 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:21.162575 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:21.162648 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 03:58:21.162798 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":678,"thread_start_us":83,"threads_started":1}
I20260927 03:58:21.176478 16718 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:58:21.177218 16718 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:58:21.180027 16718 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 766ebb4b884049e9802ba427195bb76b: Generated new TSK 0
I20260927 03:58:21.180220 16718 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.185.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab.
Entry for principal kudu/127.15.185.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab.
I20260927 03:58:21.209815 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.65:0
--local_ip_for_outbound_sockets=127.15.185.65
--webserver_interface=127.15.185.65
--webserver_port=0
--tserver_master_addrs=127.15.185.126:36197
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab
--principal=kudu/127.15.185.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:44859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:58:21.305613 16733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:21.305846 16733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:21.305888 16733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:21.307135 16733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:21.307194 16733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:21.307238 16733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.65
I20260927 03:58:21.308745 16733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:44859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab
--principal=kudu/127.15.185.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.185.65
--webserver_port=0
--tserver_master_addrs=127.15.185.126:36197
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.65
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:21.309010 16733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:21.309247 16733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:21.314594 16739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:21.314627 16741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:21.314713 16738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:21.314929 16733 server_base.cc:1034] running on GCE node
I20260927 03:58:21.315219 16733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:21.315441 16733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:21.316588 16733 hybrid_clock.cc:648] HybridClock initialized: now 1790481501316569 us; error 32 us; skew 500 ppm
Sep 27 03:58:21 dist-test-slave-glgz krb5kdc[16626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481501, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:21.319181 16733 init.cc:377] Logged in from keytab as kudu/127.15.185.65@KRBTEST.COM (short username kudu)
I20260927 03:58:21.319615 16733 webserver.cc:460] Webserver started at http://127.15.185.65:35023/ using document root <none> and password file <none>
I20260927 03:58:21.319841 16733 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:21.319907 16733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:21.320065 16733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:21.321113 16733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/minicluster-data/ts-0/data/instance:
uuid: "f56db55a17ae47bb9e8e0a7f3c5b4686"
format_stamp: "Formatted at 2026-09-27 03:58:21 on dist-test-slave-glgz"
server_key: "4d14187ef30bc2e10251cf12895f7932"
server_key_iv: "815aceafb24192468714070ffda14bc6"
server_key_version: "encryptionkey@0"
I20260927 03:58:21.321482 16733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/minicluster-data/ts-0/wal/instance:
uuid: "f56db55a17ae47bb9e8e0a7f3c5b4686"
format_stamp: "Formatted at 2026-09-27 03:58:21 on dist-test-slave-glgz"
server_key: "4d14187ef30bc2e10251cf12895f7932"
server_key_iv: "815aceafb24192468714070ffda14bc6"
server_key_version: "encryptionkey@0"
I20260927 03:58:21.322836 16733 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 03:58:21.323690 16748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:21.323868 16733 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:58:21.323964 16733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/minicluster-data/ts-0/wal
uuid: "f56db55a17ae47bb9e8e0a7f3c5b4686"
format_stamp: "Formatted at 2026-09-27 03:58:21 on dist-test-slave-glgz"
server_key: "4d14187ef30bc2e10251cf12895f7932"
server_key_iv: "815aceafb24192468714070ffda14bc6"
server_key_version: "encryptionkey@0"
I20260927 03:58:21.324051 16733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:21.340003 16733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:21.342011 16733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:21.342181 16733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:21.342432 16733 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:58:21.342751 16733 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:58:21.342806 16733 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:21.342841 16733 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:58:21.342888 16733 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:21.348635 16733 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.65:34237
I20260927 03:58:21.348742 16861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.65:34237 every 8 connection(s)
I20260927 03:58:21.349040 16733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
I20260927 03:58:21.355262 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 16733
I20260927 03:58:21.355341 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/minicluster-data/ts-0/wal/instance
Sep 27 03:58:21 dist-test-slave-glgz krb5kdc[16626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481501, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.65@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 03:58:21.355511 16101 external_mini_cluster.cc:1428] Setting key 673e3254d921e8cb287be538a3755318
I20260927 03:58:21.358865 16717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:21.349795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:44961 (local address 127.15.185.126:36197)
0927 03:58:21.349884 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:21.349887 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:21.350055 (+ 168us) server_negotiation.cc:407] Connection header received
0927 03:58:21.351182 (+ 1127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:21.351183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:21.351193 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:21.351243 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:21.351983 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:21.352476 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:21.353423 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:21.353616 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:21.353669 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:21.355901 (+ 2232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:21.355902 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:21.355903 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:21.355910 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:21.357449 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:21.357919 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:21.357921 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:21.357921 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:21.357987 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:21.358242 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:21.358244 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:21.358244 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:21.358381 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:21.358451 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:21.358694 (+ 243us) server_negotiation.cc:299] Negotiation successful
0927 03:58:21.358774 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 03:58:21.359110 16864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:21.349757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:36197 (local address 127.15.185.65:44961)
0927 03:58:21.349936 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:21.349972 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:21.350996 (+ 1024us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:21.351304 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:21.351307 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:21.351645 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:21.351908 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:21.351914 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:21.352546 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:21.352547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:21.353353 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:21.353354 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:21.353691 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:21.353692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:21.353793 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:21.354254 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:21.354262 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:21.355804 (+ 1542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:21.357565 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:21.357572 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:21.357575 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:21.357846 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:21.358055 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:21.358055 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:21.358055 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:21.358178 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:21.358444 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:21.358447 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:21.358554 (+ 107us) client_negotiation.cc:769] Sending connection context
0927 03:58:21.358669 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 03:58:21.358822 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":97,"threads_started":1}
I20260927 03:58:21.359575 16862 heartbeater.cc:347] Connected to a master server at 127.15.185.126:36197
I20260927 03:58:21.359663 16862 heartbeater.cc:464] Registering TS with master...
I20260927 03:58:21.359848 16862 heartbeater.cc:511] Master 127.15.185.126:36197 requested a full tablet report, sending...
I20260927 03:58:21.360363 16671 ts_manager.cc:194] Registered new tserver with Master: f56db55a17ae47bb9e8e0a7f3c5b4686 (127.15.185.65:34237)
I20260927 03:58:21.360960 16671 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.185.65@KRBTEST.COM'} at 127.15.185.65:44961
WARNING: no policy specified for kudu/127.15.185.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab.
Entry for principal kudu/127.15.185.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab.
I20260927 03:58:21.382104 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.66:0
--local_ip_for_outbound_sockets=127.15.185.66
--webserver_interface=127.15.185.66
--webserver_port=0
--tserver_master_addrs=127.15.185.126:36197
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab
--principal=kudu/127.15.185.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:44859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:58:21.462882 16867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:21.463150 16867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:21.463192 16867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:21.464329 16867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:21.464399 16867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:21.464445 16867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.66
I20260927 03:58:21.465901 16867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:44859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab
--principal=kudu/127.15.185.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.185.66
--webserver_port=0
--tserver_master_addrs=127.15.185.126:36197
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.66
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:21.466167 16867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:21.466390 16867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:21.471361 16875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:21.471350 16873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:21.471447 16867 server_base.cc:1034] running on GCE node
W20260927 03:58:21.471400 16872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:21.471789 16867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:21.472079 16867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:21.473250 16867 hybrid_clock.cc:648] HybridClock initialized: now 1790481501473247 us; error 37 us; skew 500 ppm
Sep 27 03:58:21 dist-test-slave-glgz krb5kdc[16626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481501, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:21.475796 16867 init.cc:377] Logged in from keytab as kudu/127.15.185.66@KRBTEST.COM (short username kudu)
I20260927 03:58:21.476264 16867 webserver.cc:460] Webserver started at http://127.15.185.66:42373/ using document root <none> and password file <none>
I20260927 03:58:21.476538 16867 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:21.476608 16867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:21.476760 16867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:21.477835 16867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/minicluster-data/ts-1/data/instance:
uuid: "19619e59a5074c89baa60cfff37e9565"
format_stamp: "Formatted at 2026-09-27 03:58:21 on dist-test-slave-glgz"
server_key: "8d1f3561af8d86a349e16d03895bd96f"
server_key_iv: "324aa76a468ef528865139013ed223a8"
server_key_version: "encryptionkey@0"
I20260927 03:58:21.478196 16867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/minicluster-data/ts-1/wal/instance:
uuid: "19619e59a5074c89baa60cfff37e9565"
format_stamp: "Formatted at 2026-09-27 03:58:21 on dist-test-slave-glgz"
server_key: "8d1f3561af8d86a349e16d03895bd96f"
server_key_iv: "324aa76a468ef528865139013ed223a8"
server_key_version: "encryptionkey@0"
I20260927 03:58:21.479621 16867 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:58:21.480376 16882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:21.480587 16867 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:58:21.480676 16867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/minicluster-data/ts-1/wal
uuid: "19619e59a5074c89baa60cfff37e9565"
format_stamp: "Formatted at 2026-09-27 03:58:21 on dist-test-slave-glgz"
server_key: "8d1f3561af8d86a349e16d03895bd96f"
server_key_iv: "324aa76a468ef528865139013ed223a8"
server_key_version: "encryptionkey@0"
I20260927 03:58:21.480759 16867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:21.490592 16867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:21.492960 16867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:21.493134 16867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:21.493347 16867 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:58:21.493652 16867 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:58:21.493706 16867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:21.493749 16867 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:58:21.493777 16867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:21.499338 16867 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.66:46689
I20260927 03:58:21.499449 16995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.66:46689 every 8 connection(s)
I20260927 03:58:21.499739 16867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
Sep 27 03:58:21 dist-test-slave-glgz krb5kdc[16626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481501, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.66@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 03:58:21.506937 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 16867
I20260927 03:58:21.507074 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/minicluster-data/ts-1/wal/instance
I20260927 03:58:21.507287 16101 external_mini_cluster.cc:1428] Setting key a7351f4b85a7ac8963cb4729a371f345
I20260927 03:58:21.509373 16717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:21.500495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:57533 (local address 127.15.185.126:36197)
0927 03:58:21.500546 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:21.500549 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:21.500694 (+ 145us) server_negotiation.cc:407] Connection header received
0927 03:58:21.501614 (+ 920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:21.501616 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:21.501628 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:21.501696 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:21.502381 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:21.502873 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:21.503973 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:21.504182 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:21.504249 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:21.506258 (+ 2009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:21.506259 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:21.506259 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:21.506266 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:21.507882 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:21.508386 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:21.508387 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:21.508388 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:21.508449 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:21.508697 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:21.508698 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:21.508698 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:21.508816 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:21.508893 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:21.509191 (+ 298us) server_negotiation.cc:299] Negotiation successful
0927 03:58:21.509288 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 03:58:21.509660 16998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:21.500415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:36197 (local address 127.15.185.66:57533)
0927 03:58:21.500577 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:21.500620 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:21.501507 (+ 887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:21.501772 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:21.501775 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:21.502072 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:21.502276 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:21.502280 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:21.502955 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:21.502956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:21.503846 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:21.503848 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:21.504265 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:21.504266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:21.504379 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:21.504840 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:21.504847 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:21.506175 (+ 1328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:21.508052 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:21.508059 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:21.508061 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:21.508293 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:21.508517 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:21.508518 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:21.508518 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:21.508627 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:21.508884 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:21.508890 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:21.509026 (+ 136us) client_negotiation.cc:769] Sending connection context
0927 03:58:21.509147 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 03:58:21.509336 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":92,"threads_started":1}
I20260927 03:58:21.510193 16996 heartbeater.cc:347] Connected to a master server at 127.15.185.126:36197
I20260927 03:58:21.510288 16996 heartbeater.cc:464] Registering TS with master...
I20260927 03:58:21.510483 16996 heartbeater.cc:511] Master 127.15.185.126:36197 requested a full tablet report, sending...
I20260927 03:58:21.510883 16671 ts_manager.cc:194] Registered new tserver with Master: 19619e59a5074c89baa60cfff37e9565 (127.15.185.66:46689)
I20260927 03:58:21.511327 16671 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.185.66@KRBTEST.COM'} at 127.15.185.66:57533
WARNING: no policy specified for kudu/127.15.185.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab.
Entry for principal kudu/127.15.185.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab.
I20260927 03:58:21.534834 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.67:0
--local_ip_for_outbound_sockets=127.15.185.67
--webserver_interface=127.15.185.67
--webserver_port=0
--tserver_master_addrs=127.15.185.126:36197
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab
--principal=kudu/127.15.185.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:44859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:58:21.614209 17001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:21.614447 17001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:21.614483 17001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:21.615658 17001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:21.615724 17001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:21.615768 17001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.67
I20260927 03:58:21.617224 17001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:44859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab
--principal=kudu/127.15.185.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.185.67
--webserver_port=0
--tserver_master_addrs=127.15.185.126:36197
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.67
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:21.617479 17001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:21.617704 17001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:21.623021 17007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:21.622997 17006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:21.622997 17009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:21.623145 17001 server_base.cc:1034] running on GCE node
I20260927 03:58:21.623457 17001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:21.623673 17001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:21.624814 17001 hybrid_clock.cc:648] HybridClock initialized: now 1790481501624804 us; error 36 us; skew 500 ppm
Sep 27 03:58:21 dist-test-slave-glgz krb5kdc[16626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481501, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:21.627339 17001 init.cc:377] Logged in from keytab as kudu/127.15.185.67@KRBTEST.COM (short username kudu)
I20260927 03:58:21.627820 17001 webserver.cc:460] Webserver started at http://127.15.185.67:34319/ using document root <none> and password file <none>
I20260927 03:58:21.628072 17001 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:21.628141 17001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:21.628274 17001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:21.629309 17001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/minicluster-data/ts-2/data/instance:
uuid: "f22298d9455d4518b25258052156d235"
format_stamp: "Formatted at 2026-09-27 03:58:21 on dist-test-slave-glgz"
server_key: "7dc3d1c4828ba2f23348e43a511857cf"
server_key_iv: "b67d874a37be6c9989f25c7d9a908c5b"
server_key_version: "encryptionkey@0"
I20260927 03:58:21.629727 17001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/minicluster-data/ts-2/wal/instance:
uuid: "f22298d9455d4518b25258052156d235"
format_stamp: "Formatted at 2026-09-27 03:58:21 on dist-test-slave-glgz"
server_key: "7dc3d1c4828ba2f23348e43a511857cf"
server_key_iv: "b67d874a37be6c9989f25c7d9a908c5b"
server_key_version: "encryptionkey@0"
I20260927 03:58:21.631148 17001 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 03:58:21.631884 17016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:21.632066 17001 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:58:21.632165 17001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/minicluster-data/ts-2/wal
uuid: "f22298d9455d4518b25258052156d235"
format_stamp: "Formatted at 2026-09-27 03:58:21 on dist-test-slave-glgz"
server_key: "7dc3d1c4828ba2f23348e43a511857cf"
server_key_iv: "b67d874a37be6c9989f25c7d9a908c5b"
server_key_version: "encryptionkey@0"
I20260927 03:58:21.632251 17001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:21.667936 17001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:21.669788 17001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:21.669948 17001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:21.670188 17001 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:58:21.670490 17001 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:58:21.670543 17001 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:21.670595 17001 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:58:21.670627 17001 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:21.676298 17001 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.67:43517
I20260927 03:58:21.676337 17129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.67:43517 every 8 connection(s)
I20260927 03:58:21.676673 17001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
I20260927 03:58:21.679512 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 17001
I20260927 03:58:21.679622 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481498018407-16101-0/minicluster-data/ts-2/wal/instance
I20260927 03:58:21.679847 16101 external_mini_cluster.cc:1428] Setting key 57e9fbeea8a188d81962ce107b327de5
Sep 27 03:58:21 dist-test-slave-glgz krb5kdc[16626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481501, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.67@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 03:58:21.686733 16717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:21.677425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:38295 (local address 127.15.185.126:36197)
0927 03:58:21.677488 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:21.677490 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:21.677738 (+ 248us) server_negotiation.cc:407] Connection header received
0927 03:58:21.678684 (+ 946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:21.678685 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:21.678697 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:21.678755 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:21.679440 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:21.680075 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:21.681432 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:21.681617 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:21.681665 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:21.684134 (+ 2469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:21.684136 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:21.684136 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:21.684142 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:21.685503 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:21.685936 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:21.685937 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:21.685937 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:21.685984 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:21.686196 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:21.686196 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:21.686196 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:21.686290 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:21.686340 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:21.686612 (+ 272us) server_negotiation.cc:299] Negotiation successful
0927 03:58:21.686670 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 03:58:21.686924 17132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:21.677419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:36197 (local address 127.15.185.67:38295)
0927 03:58:21.677616 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:21.677653 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:21.678564 (+ 911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:21.678829 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:21.678832 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:21.679146 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:21.679339 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:21.679344 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:21.680187 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:21.680188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:21.681353 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:21.681355 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:21.681686 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:21.681686 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:21.681789 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:21.682295 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:21.682304 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:21.684043 (+ 1739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:21.685586 (+ 1543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:21.685591 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:21.685593 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:21.685816 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:21.686041 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:21.686042 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:21.686042 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:21.686142 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:21.686368 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:21.686374 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:21.686492 (+ 118us) client_negotiation.cc:769] Sending connection context
0927 03:58:21.686587 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 03:58:21.686704 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":103,"threads_started":1}
I20260927 03:58:21.687492 17130 heartbeater.cc:347] Connected to a master server at 127.15.185.126:36197
I20260927 03:58:21.687597 17130 heartbeater.cc:464] Registering TS with master...
I20260927 03:58:21.687776 17130 heartbeater.cc:511] Master 127.15.185.126:36197 requested a full tablet report, sending...
I20260927 03:58:21.688130 16671 ts_manager.cc:194] Registered new tserver with Master: f22298d9455d4518b25258052156d235 (127.15.185.67:43517)
I20260927 03:58:21.688488 16671 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.185.67@KRBTEST.COM'} at 127.15.185.67:38295
I20260927 03:58:21.691588 16101 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:58:21.698807 16717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:21.693042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56522 (local address 127.15.185.126:36197)
0927 03:58:21.693104 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:21.693106 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:21.693194 (+ 88us) server_negotiation.cc:407] Connection header received
0927 03:58:21.693270 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:21.693271 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:21.693280 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:21.693335 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:21.693869 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:21.694331 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:21.695087 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:21.695287 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:21.695369 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:21.696523 (+ 1154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:21.696525 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:21.696525 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:21.696531 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:21.697679 (+ 1148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:21.698020 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:21.698021 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:21.698022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:21.698079 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:21.698277 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:21.698278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:21.698278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:21.698423 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:21.698523 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:21.698635 (+ 112us) server_negotiation.cc:299] Negotiation successful
0927 03:58:21.698709 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 03:58:21.700723 16671 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56522:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:58:21.702244 16671 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:58:21.708439 17142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:21.704438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.66:46689 (local address 127.0.0.1:45056)
0927 03:58:21.704741 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:21.704749 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:21.704789 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:21.705040 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:21.705043 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:21.705045 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:58:21.705113 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:21.705115 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:21.706115 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:21.706116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:21.707287 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:21.707289 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:21.708151 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:21.708151 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:21.708229 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:21.708243 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 03:58:21.708317 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 03:58:21.708365 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":75,"threads_started":1}
I20260927 03:58:21.708495 17145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:21.704681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45056 (local address 127.15.185.66:46689)
0927 03:58:21.704854 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:21.704860 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:21.704869 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:58:21.704908 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:21.704910 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:21.704924 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:21.705029 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:58:21.705199 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:21.706012 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:21.707402 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:21.708060 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:21.708137 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:21.708194 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:21.708345 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 03:58:21.708387 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":77,"threads_started":1}
I20260927 03:58:21.708725 17141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:21.704421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.67:43517 (local address 127.0.0.1:43264)
0927 03:58:21.704528 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:21.704569 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:21.704629 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:21.705240 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:21.705241 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:21.705244 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:58:21.705299 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:21.705300 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:21.706582 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:21.706583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:21.707595 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:21.707597 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:21.708495 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:21.708496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:21.708565 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:21.708570 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:58:21.708604 (+ 34us) client_negotiation.cc:240] Negotiation successful
0927 03:58:21.708646 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":40,"threads_started":1}
I20260927 03:58:21.709035 17143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:21.704538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43264 (local address 127.15.185.67:43517)
0927 03:58:21.705079 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:21.705082 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:21.705091 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:58:21.705124 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:21.705126 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:21.705142 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:21.705373 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:58:21.705451 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:21.706471 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:21.707683 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:21.708397 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:21.708833 (+ 436us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:21.708899 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:21.708924 (+ 25us) server_negotiation.cc:299] Negotiation successful
0927 03:58:21.708967 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":92,"threads_started":1}
I20260927 03:58:21.709316 16930 tablet_service.cc:1467] Processing CreateTablet for tablet 334dc146bead4fcc962e5d990d5a09b5 (DEFAULT_TABLE table=test-table [id=a216bff29c004a6b9bec79997f022223]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:58:21.709654 16930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 334dc146bead4fcc962e5d990d5a09b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:21.709767 17144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:21.704642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.65:34237 (local address 127.0.0.1:41606)
0927 03:58:21.704769 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:21.704780 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:21.704837 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:21.705268 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:21.705269 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:21.705271 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:58:21.705309 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:21.705311 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:21.707675 (+ 2364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:21.707676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:21.708471 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:21.708473 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:21.709532 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:21.709533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:21.709597 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:21.709602 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:58:21.709665 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 03:58:21.709705 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":2,"thread_start_us":82,"threads_started":1}
I20260927 03:58:21.709824 17146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:21.704866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41606 (local address 127.15.185.65:34237)
0927 03:58:21.705025 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:21.705030 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:21.705042 (+ 12us) server_negotiation.cc:407] Connection header received
0927 03:58:21.705091 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:21.705093 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:21.705109 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:21.705190 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:58:21.705388 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:21.706484 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:21.708724 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:21.709441 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:21.709514 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:21.709565 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:21.709696 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 03:58:21.709741 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":97,"threads_started":1}
I20260927 03:58:21.709761 17064 tablet_service.cc:1467] Processing CreateTablet for tablet 334dc146bead4fcc962e5d990d5a09b5 (DEFAULT_TABLE table=test-table [id=a216bff29c004a6b9bec79997f022223]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:58:21.710054 17064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 334dc146bead4fcc962e5d990d5a09b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:21.712404 17147 tablet_bootstrap.cc:492] T 334dc146bead4fcc962e5d990d5a09b5 P 19619e59a5074c89baa60cfff37e9565: Bootstrap starting.
I20260927 03:58:21.712677 17148 tablet_bootstrap.cc:492] T 334dc146bead4fcc962e5d990d5a09b5 P f22298d9455d4518b25258052156d235: Bootstrap starting.
I20260927 03:58:21.713056 17147 tablet_bootstrap.cc:654] T 334dc146bead4fcc962e5d990d5a09b5 P 19619e59a5074c89baa60cfff37e9565: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:21.713122 16796 tablet_service.cc:1467] Processing CreateTablet for tablet 334dc146bead4fcc962e5d990d5a09b5 (DEFAULT_TABLE table=test-table [id=a216bff29c004a6b9bec79997f022223]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:58:21.713299 17148 tablet_bootstrap.cc:654] T 334dc146bead4fcc962e5d990d5a09b5 P f22298d9455d4518b25258052156d235: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:21.713475 16796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 334dc146bead4fcc962e5d990d5a09b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:21.713645 17147 log.cc:824] T 334dc146bead4fcc962e5d990d5a09b5 P 19619e59a5074c89baa60cfff37e9565: Log is configured to *not* fsync() on all Append() calls
I20260927 03:58:21.713830 17148 log.cc:824] T 334dc146bead4fcc962e5d990d5a09b5 P f22298d9455d4518b25258052156d235: Log is configured to *not* fsync() on all Append() calls
I20260927 03:58:21.714399 17147 tablet_bootstrap.cc:492] T 334dc146bead4fcc962e5d990d5a09b5 P 19619e59a5074c89baa60cfff37e9565: No bootstrap required, opened a new log
I20260927 03:58:21.714481 17147 ts_tablet_manager.cc:1397] T 334dc146bead4fcc962e5d990d5a09b5 P 19619e59a5074c89baa60cfff37e9565: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:58:21.714666 17148 tablet_bootstrap.cc:492] T 334dc146bead4fcc962e5d990d5a09b5 P f22298d9455d4518b25258052156d235: No bootstrap required, opened a new log
I20260927 03:58:21.714730 17148 ts_tablet_manager.cc:1397] T 334dc146bead4fcc962e5d990d5a09b5 P f22298d9455d4518b25258052156d235: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:58:21.716296 17151 tablet_bootstrap.cc:492] T 334dc146bead4fcc962e5d990d5a09b5 P f56db55a17ae47bb9e8e0a7f3c5b4686: Bootstrap starting.
I20260927 03:58:21.716501 17147 raft_consensus.cc:348] T 334dc146bead4fcc962e5d990d5a09b5 P 19619e59a5074c89baa60cfff37e9565 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f22298d9455d4518b25258052156d235" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 43517 } } peers { permanent_uuid: "19619e59a5074c89baa60cfff37e9565" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 46689 } } peers { permanent_uuid: "f56db55a17ae47bb9e8e0a7f3c5b4686" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 34237 } }
I20260927 03:58:21.716689 17147 raft_consensus.cc:374] T 334dc146bead4fcc962e5d990d5a09b5 P 19619e59a5074c89baa60cfff37e9565 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:21.716614 17148 raft_consensus.cc:348] T 334dc146bead4fcc962e5d990d5a09b5 P f22298d9455d4518b25258052156d235 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f22298d9455d4518b25258052156d235" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 43517 } } peers { permanent_uuid: "19619e59a5074c89baa60cfff37e9565" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 46689 } } peers { permanent_uuid: "f56db55a17ae47bb9e8e0a7f3c5b4686" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 34237 } }
I20260927 03:58:21.716727 17147 raft_consensus.cc:729] T 334dc146bead4fcc962e5d990d5a09b5 P 19619e59a5074c89baa60cfff37e9565 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19619e59a5074c89baa60cfff37e9565, State: Initialized, Role: FOLLOWER
I20260927 03:58:21.716756 17148 raft_consensus.cc:374] T 334dc146bead4fcc962e5d990d5a09b5 P f22298d9455d4518b25258052156d235 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:21.716787 17148 raft_consensus.cc:729] T 334dc146bead4fcc962e5d990d5a09b5 P f22298d9455d4518b25258052156d235 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f22298d9455d4518b25258052156d235, State: Initialized, Role: FOLLOWER
I20260927 03:58:21.716941 17151 tablet_bootstrap.cc:654] T 334dc146bead4fcc962e5d990d5a09b5 P f56db55a17ae47bb9e8e0a7f3c5b4686: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:21.716889 17148 consensus_queue.cc:260] T 334dc146bead4fcc962e5d990d5a09b5 P f22298d9455d4518b25258052156d235 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f22298d9455d4518b25258052156d235" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 43517 } } peers { permanent_uuid: "19619e59a5074c89baa60cfff37e9565" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 46689 } } peers { permanent_uuid: "f56db55a17ae47bb9e8e0a7f3c5b4686" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 34237 } }
I20260927 03:58:21.716883 17147 consensus_queue.cc:260] T 334dc146bead4fcc962e5d990d5a09b5 P 19619e59a5074c89baa60cfff37e9565 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f22298d9455d4518b25258052156d235" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 43517 } } peers { permanent_uuid: "19619e59a5074c89baa60cfff37e9565" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 46689 } } peers { permanent_uuid: "f56db55a17ae47bb9e8e0a7f3c5b4686" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 34237 } }
I20260927 03:58:21.717195 17148 ts_tablet_manager.cc:1428] T 334dc146bead4fcc962e5d990d5a09b5 P f22298d9455d4518b25258052156d235: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 03:58:21.717195 17147 ts_tablet_manager.cc:1428] T 334dc146bead4fcc962e5d990d5a09b5 P 19619e59a5074c89baa60cfff37e9565: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:58:21.717317 17130 heartbeater.cc:503] Master 127.15.185.126:36197 was elected leader, sending a full tablet report...
I20260927 03:58:21.717337 16996 heartbeater.cc:503] Master 127.15.185.126:36197 was elected leader, sending a full tablet report...
I20260927 03:58:21.717589 17151 log.cc:824] T 334dc146bead4fcc962e5d990d5a09b5 P f56db55a17ae47bb9e8e0a7f3c5b4686: Log is configured to *not* fsync() on all Append() calls
I20260927 03:58:21.718479 17151 tablet_bootstrap.cc:492] T 334dc146bead4fcc962e5d990d5a09b5 P f56db55a17ae47bb9e8e0a7f3c5b4686: No bootstrap required, opened a new log
I20260927 03:58:21.718544 17151 ts_tablet_manager.cc:1397] T 334dc146bead4fcc962e5d990d5a09b5 P f56db55a17ae47bb9e8e0a7f3c5b4686: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:58:21.719893 17151 raft_consensus.cc:348] T 334dc146bead4fcc962e5d990d5a09b5 P f56db55a17ae47bb9e8e0a7f3c5b4686 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f22298d9455d4518b25258052156d235" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 43517 } } peers { permanent_uuid: "19619e59a5074c89baa60cfff37e9565" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 46689 } } peers { permanent_uuid: "f56db55a17ae47bb9e8e0a7f3c5b4686" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 34237 } }
I20260927 03:58:21.720046 17151 raft_consensus.cc:374] T 334dc146bead4fcc962e5d990d5a09b5 P f56db55a17ae47bb9e8e0a7f3c5b4686 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:21.720096 17151 raft_consensus.cc:729] T 334dc146bead4fcc962e5d990d5a09b5 P f56db55a17ae47bb9e8e0a7f3c5b4686 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f56db55a17ae47bb9e8e0a7f3c5b4686, State: Initialized, Role: FOLLOWER
I20260927 03:58:21.720228 17151 consensus_queue.cc:260] T 334dc146bead4fcc962e5d990d5a09b5 P f56db55a17ae47bb9e8e0a7f3c5b4686 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f22298d9455d4518b25258052156d235" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 43517 } } peers { permanent_uuid: "19619e59a5074c89baa60cfff37e9565" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 46689 } } peers { permanent_uuid: "f56db55a17ae47bb9e8e0a7f3c5b4686" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 34237 } }
I20260927 03:58:21.720484 17151 ts_tablet_manager.cc:1428] T 334dc146bead4fcc962e5d990d5a09b5 P f56db55a17ae47bb9e8e0a7f3c5b4686: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:58:21.720491 16862 heartbeater.cc:503] Master 127.15.185.126:36197 was elected leader, sending a full tablet report...
W20260927 03:58:21.750304 16997 tablet.cc:2369] T 334dc146bead4fcc962e5d990d5a09b5 P 19619e59a5074c89baa60cfff37e9565: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 03:58:21.849761 16863 tablet.cc:2369] T 334dc146bead4fcc962e5d990d5a09b5 P f56db55a17ae47bb9e8e0a7f3c5b4686: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:58:21.926329 17152 raft_consensus.cc:482] T 334dc146bead4fcc962e5d990d5a09b5 P f22298d9455d4518b25258052156d235 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:58:21.926491 17152 raft_consensus.cc:504] T 334dc146bead4fcc962e5d990d5a09b5 P f22298d9455d4518b25258052156d235 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f22298d9455d4518b25258052156d235" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 43517 } } peers { permanent_uuid: "19619e59a5074c89baa60cfff37e9565" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 46689 } } peers { permanent_uuid: "f56db55a17ae47bb9e8e0a7f3c5b4686" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 34237 } }
I20260927 03:58:21.926895 17152 leader_election.cc:288] T 334dc146bead4fcc962e5d990d5a09b5 P f22298d9455d4518b25258052156d235 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19619e59a5074c89baa60cfff37e9565 (127.15.185.66:46689), f56db55a17ae47bb9e8e0a7f3c5b4686 (127.15.185.65:34237)
W20260927 03:58:21.927186 17131 tablet.cc:2369] T 334dc146bead4fcc962e5d990d5a09b5 P f22298d9455d4518b25258052156d235: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:58:21.930868 17145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:21.927221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:59173 (local address 127.15.185.66:46689)
0927 03:58:21.927270 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:21.927273 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:21.927391 (+ 118us) server_negotiation.cc:407] Connection header received
0927 03:58:21.927425 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:21.927426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:21.927435 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:21.927533 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:58:21.927747 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:21.928534 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:21.930095 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:21.930496 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:21.930554 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:21.930561 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:21.930754 (+ 193us) server_negotiation.cc:299] Negotiation successful
0927 03:58:21.930792 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 03:58:21.930851 17132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:21.927178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.66:46689 (local address 127.15.185.67:59173)
0927 03:58:21.927237 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:21.927250 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:21.927323 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:21.927540 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:21.927542 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:21.927544 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:58:21.927681 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:21.927684 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:21.928624 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:21.928625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:21.929967 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:21.929970 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:21.930584 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:21.930585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:21.930675 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:21.930679 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:58:21.930735 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 03:58:21.930780 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":73}
I20260927 03:58:21.931023 17156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:21.927293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.65:34237 (local address 127.15.185.67:34031)
0927 03:58:21.927456 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:21.927466 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:21.927520 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:21.927643 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:21.927644 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:21.927645 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:58:21.927727 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:21.927729 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:21.928624 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:21.928625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:21.929967 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:21.929970 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:21.930765 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:21.930767 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:21.930835 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:21.930839 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:58:21.930898 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 03:58:21.930940 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":68,"thread_start_us":92,"threads_started":1}
I20260927 03:58:21.931077 17146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:21.927460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:34031 (local address 127.15.185.65:34237)
0927 03:58:21.927526 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:21.927528 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:21.927540 (+ 12us) server_negotiation.cc:407] Connection header received
0927 03:58:21.927567 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:21.927568 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:21.927576 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:21.927626 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:58:21.927793 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:21.928536 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:21.930132 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:21.930661 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:21.930741 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:21.930749 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:21.930919 (+ 170us) server_negotiation.cc:299] Negotiation successful
0927 03:58:21.930966 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 03:58:21.931249 16950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "334dc146bead4fcc962e5d990d5a09b5" candidate_uuid: "f22298d9455d4518b25258052156d235" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19619e59a5074c89baa60cfff37e9565" is_pre_election: true
I20260927 03:58:21.931403 16950 raft_consensus.cc:2454] T 334dc146bead4fcc962e5d990d5a09b5 P 19619e59a5074c89baa60cfff37e9565 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f22298d9455d4518b25258052156d235 in term 0.
I20260927 03:58:21.931573 16816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "334dc146bead4fcc962e5d990d5a09b5" candidate_uuid: "f22298d9455d4518b25258052156d235" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f56db55a17ae47bb9e8e0a7f3c5b4686" is_pre_election: true
I20260927 03:58:21.931679 17017 leader_election.cc:302] T 334dc146bead4fcc962e5d990d5a09b5 P f22298d9455d4518b25258052156d235 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19619e59a5074c89baa60cfff37e9565, f22298d9455d4518b25258052156d235; no voters:
I20260927 03:58:21.931720 16816 raft_consensus.cc:2454] T 334dc146bead4fcc962e5d990d5a09b5 P f56db55a17ae47bb9e8e0a7f3c5b4686 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f22298d9455d4518b25258052156d235 in term 0.
I20260927 03:58:21.931829 17152 raft_consensus.cc:2781] T 334dc146bead4fcc962e5d990d5a09b5 P f22298d9455d4518b25258052156d235 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:58:21.931879 17152 raft_consensus.cc:482] T 334dc146bead4fcc962e5d990d5a09b5 P f22298d9455d4518b25258052156d235 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:58:21.931919 17152 raft_consensus.cc:3037] T 334dc146bead4fcc962e5d990d5a09b5 P f22298d9455d4518b25258052156d235 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:21.932628 17152 raft_consensus.cc:504] T 334dc146bead4fcc962e5d990d5a09b5 P f22298d9455d4518b25258052156d235 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f22298d9455d4518b25258052156d235" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 43517 } } peers { permanent_uuid: "19619e59a5074c89baa60cfff37e9565" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 46689 } } peers { permanent_uuid: "f56db55a17ae47bb9e8e0a7f3c5b4686" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 34237 } }
I20260927 03:58:21.932729 17152 leader_election.cc:288] T 334dc146bead4fcc962e5d990d5a09b5 P f22298d9455d4518b25258052156d235 [CANDIDATE]: Term 1 election: Requested vote from peers 19619e59a5074c89baa60cfff37e9565 (127.15.185.66:46689), f56db55a17ae47bb9e8e0a7f3c5b4686 (127.15.185.65:34237)
I20260927 03:58:21.932897 16950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "334dc146bead4fcc962e5d990d5a09b5" candidate_uuid: "f22298d9455d4518b25258052156d235" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19619e59a5074c89baa60cfff37e9565"
I20260927 03:58:21.932940 16816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "334dc146bead4fcc962e5d990d5a09b5" candidate_uuid: "f22298d9455d4518b25258052156d235" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f56db55a17ae47bb9e8e0a7f3c5b4686"
I20260927 03:58:21.933003 16816 raft_consensus.cc:3037] T 334dc146bead4fcc962e5d990d5a09b5 P f56db55a17ae47bb9e8e0a7f3c5b4686 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:21.932982 16950 raft_consensus.cc:3037] T 334dc146bead4fcc962e5d990d5a09b5 P 19619e59a5074c89baa60cfff37e9565 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:21.933714 16816 raft_consensus.cc:2454] T 334dc146bead4fcc962e5d990d5a09b5 P f56db55a17ae47bb9e8e0a7f3c5b4686 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f22298d9455d4518b25258052156d235 in term 1.
I20260927 03:58:21.933825 16950 raft_consensus.cc:2454] T 334dc146bead4fcc962e5d990d5a09b5 P 19619e59a5074c89baa60cfff37e9565 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f22298d9455d4518b25258052156d235 in term 1.
I20260927 03:58:21.933889 17017 leader_election.cc:302] T 334dc146bead4fcc962e5d990d5a09b5 P f22298d9455d4518b25258052156d235 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f22298d9455d4518b25258052156d235, f56db55a17ae47bb9e8e0a7f3c5b4686; no voters:
I20260927 03:58:21.933961 17152 raft_consensus.cc:2781] T 334dc146bead4fcc962e5d990d5a09b5 P f22298d9455d4518b25258052156d235 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:58:21.934129 17152 raft_consensus.cc:686] T 334dc146bead4fcc962e5d990d5a09b5 P f22298d9455d4518b25258052156d235 [term 1 LEADER]: Becoming Leader. State: Replica: f22298d9455d4518b25258052156d235, State: Running, Role: LEADER
I20260927 03:58:21.934232 17152 consensus_queue.cc:237] T 334dc146bead4fcc962e5d990d5a09b5 P f22298d9455d4518b25258052156d235 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f22298d9455d4518b25258052156d235" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 43517 } } peers { permanent_uuid: "19619e59a5074c89baa60cfff37e9565" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 46689 } } peers { permanent_uuid: "f56db55a17ae47bb9e8e0a7f3c5b4686" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 34237 } }
I20260927 03:58:21.935038 16670 catalog_manager.cc:5495] T 334dc146bead4fcc962e5d990d5a09b5 P f22298d9455d4518b25258052156d235 reported cstate change: term changed from 0 to 1, leader changed from <none> to f22298d9455d4518b25258052156d235 (127.15.185.67). New cstate: current_term: 1 leader_uuid: "f22298d9455d4518b25258052156d235" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f22298d9455d4518b25258052156d235" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 43517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19619e59a5074c89baa60cfff37e9565" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 46689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f56db55a17ae47bb9e8e0a7f3c5b4686" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 34237 } health_report { overall_health: UNKNOWN } } }
Sep 27 03:58:21 dist-test-slave-glgz krb5kdc[16626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.185.65@KRBTEST.COM
I20260927 03:58:21.996663 17146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:21.989121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41620 (local address 127.15.185.65:34237)
0927 03:58:21.989187 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:21.989189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:21.989275 (+ 86us) server_negotiation.cc:407] Connection header received
0927 03:58:21.989319 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:21.989320 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:21.989329 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:21.989383 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:21.989945 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:21.990412 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:21.991171 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:21.991344 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:21.991397 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:21.993903 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:21.993915 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:21.993916 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:21.993922 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:21.995611 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:21.995955 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:21.995958 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:21.995958 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:21.996019 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:21.996209 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:21.996211 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:21.996211 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:21.996356 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:21.996465 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:21.996525 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 03:58:21.996596 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Sep 27 03:58:22 dist-test-slave-glgz krb5kdc[16626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481502, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 03:58:22 dist-test-slave-glgz krb5kdc[16626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481502, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.185.65@KRBTEST.COM
I20260927 03:58:22.019778 17146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:22.012661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41622 (local address 127.15.185.65:34237)
0927 03:58:22.012756 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:22.012758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:22.012805 (+ 47us) server_negotiation.cc:407] Connection header received
0927 03:58:22.012851 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:22.012852 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:22.012861 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:22.012926 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:22.013474 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:22.013985 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:22.014700 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:22.014892 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:22.014958 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:22.017201 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:22.017203 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:22.017203 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:22.017209 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:22.018717 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:22.019017 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:22.019018 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:22.019018 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:22.019120 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:22.019286 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:22.019286 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:22.019286 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:22.019388 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:22.019507 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:22.019617 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 03:58:22.019677 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
Sep 27 03:58:22 dist-test-slave-glgz krb5kdc[16626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 03:58:22 dist-test-slave-glgz krb5kdc[16626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.185.65@KRBTEST.COM
I20260927 03:58:22.042586 17146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:22.036058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41626 (local address 127.15.185.65:34237)
0927 03:58:22.036136 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:22.036138 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:22.036209 (+ 71us) server_negotiation.cc:407] Connection header received
0927 03:58:22.036280 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:22.036281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:22.036290 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:22.036341 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:22.036882 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:22.037354 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:22.038056 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:22.038230 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:22.038284 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:22.040459 (+ 2175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:22.040461 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:22.040461 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:22.040468 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:22.041702 (+ 1234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:22.041993 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:22.041994 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:22.041994 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:22.042042 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:22.042205 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:22.042205 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:22.042206 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:22.042290 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:22.042360 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:22.042442 (+ 82us) server_negotiation.cc:299] Negotiation successful
0927 03:58:22.042493 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260927 03:58:22.043154 16776 tablet_service.cc:2908] Rejecting scan request for tablet 334dc146bead4fcc962e5d990d5a09b5: Uninitialized: clean time has not yet been initialized
I20260927 03:58:22.044193 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 16733
I20260927 03:58:22.049606 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 16867
I20260927 03:58:22.054418 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 17001
I20260927 03:58:22.059016 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 16639
2026-09-27T03:58:22Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2181 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:58:22 dist-test-slave-glgz krb5kdc[17171](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:58:22 dist-test-slave-glgz krb5kdc[17171](info): set up 2 sockets
Sep 27 03:58:22 dist-test-slave-glgz krb5kdc[17171](info): commencing operation
krb5kdc: starting...
W20260927 03:58:23.095033 16101 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Sep 27 03:58:23 dist-test-slave-glgz krb5kdc[17171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:58:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:58:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.185.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab.
Entry for principal kudu/127.15.185.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab.
I20260927 03:58:23.222393 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:44897
--webserver_interface=127.15.185.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:35381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.185.126:44897
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:58:23.311874 17184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:23.312090 17184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:23.312114 17184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:23.313283 17184 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:58:23.313323 17184 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 03:58:23.313336 17184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:23.313350 17184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:23.313390 17184 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:58:23.313416 17184 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:58:23.314889 17184 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:35381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.185.126:44897
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:44897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:23.315191 17184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:23.315438 17184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:23.320426 17190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:23.320571 17189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:23.320688 17184 server_base.cc:1034] running on GCE node
W20260927 03:58:23.320595 17192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:23.321054 17184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:23.321368 17184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:23.322552 17184 hybrid_clock.cc:648] HybridClock initialized: now 1790481503322553 us; error 44 us; skew 500 ppm
Sep 27 03:58:23 dist-test-slave-glgz krb5kdc[17171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481503, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:23.331897 17184 init.cc:377] Logged in from keytab as kudu/127.15.185.126@KRBTEST.COM (short username kudu)
I20260927 03:58:23.332368 17184 webserver.cc:460] Webserver started at http://127.15.185.126:38307/ using document root <none> and password file <none>
I20260927 03:58:23.332638 17184 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:23.332713 17184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:23.332867 17184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:23.333999 17184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/master-0/data/instance:
uuid: "b08185cf201e47eebf83a05b40d72b4d"
format_stamp: "Formatted at 2026-09-27 03:58:23 on dist-test-slave-glgz"
server_key: "b0bee52b886daf537377bd6eeda52529"
server_key_iv: "741fed7eb37f762b4030b9cc32e45de9"
server_key_version: "encryptionkey@0"
I20260927 03:58:23.334372 17184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/master-0/wal/instance:
uuid: "b08185cf201e47eebf83a05b40d72b4d"
format_stamp: "Formatted at 2026-09-27 03:58:23 on dist-test-slave-glgz"
server_key: "b0bee52b886daf537377bd6eeda52529"
server_key_iv: "741fed7eb37f762b4030b9cc32e45de9"
server_key_version: "encryptionkey@0"
I20260927 03:58:23.335855 17184 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:58:23.336614 17199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:23.336848 17184 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:58:23.336952 17184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/master-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/master-0/wal
uuid: "b08185cf201e47eebf83a05b40d72b4d"
format_stamp: "Formatted at 2026-09-27 03:58:23 on dist-test-slave-glgz"
server_key: "b0bee52b886daf537377bd6eeda52529"
server_key_iv: "741fed7eb37f762b4030b9cc32e45de9"
server_key_version: "encryptionkey@0"
I20260927 03:58:23.337003 17184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:23.351830 17184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:23.354194 17184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:23.354305 17184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:23.358906 17184 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.126:44897
I20260927 03:58:23.358963 17261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.126:44897 every 8 connection(s)
I20260927 03:58:23.359285 17184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
I20260927 03:58:23.359930 17262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:23.362156 17262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b08185cf201e47eebf83a05b40d72b4d: Bootstrap starting.
I20260927 03:58:23.363181 17262 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b08185cf201e47eebf83a05b40d72b4d: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:23.363598 17262 log.cc:824] T 00000000000000000000000000000000 P b08185cf201e47eebf83a05b40d72b4d: Log is configured to *not* fsync() on all Append() calls
I20260927 03:58:23.364388 17262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b08185cf201e47eebf83a05b40d72b4d: No bootstrap required, opened a new log
I20260927 03:58:23.366498 17262 raft_consensus.cc:348] T 00000000000000000000000000000000 P b08185cf201e47eebf83a05b40d72b4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b08185cf201e47eebf83a05b40d72b4d" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 44897 } }
I20260927 03:58:23.366662 17262 raft_consensus.cc:374] T 00000000000000000000000000000000 P b08185cf201e47eebf83a05b40d72b4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:23.366707 17262 raft_consensus.cc:729] T 00000000000000000000000000000000 P b08185cf201e47eebf83a05b40d72b4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b08185cf201e47eebf83a05b40d72b4d, State: Initialized, Role: FOLLOWER
I20260927 03:58:23.366878 17262 consensus_queue.cc:260] T 00000000000000000000000000000000 P b08185cf201e47eebf83a05b40d72b4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b08185cf201e47eebf83a05b40d72b4d" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 44897 } }
I20260927 03:58:23.366981 17262 raft_consensus.cc:388] T 00000000000000000000000000000000 P b08185cf201e47eebf83a05b40d72b4d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:58:23.367028 17262 raft_consensus.cc:482] T 00000000000000000000000000000000 P b08185cf201e47eebf83a05b40d72b4d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:58:23.367125 17262 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b08185cf201e47eebf83a05b40d72b4d [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:23.367415 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 17184
I20260927 03:58:23.367492 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/master-0/wal/instance
I20260927 03:58:23.367667 16101 external_mini_cluster.cc:1428] Setting key 9a94cf01a2478579595d9744c78f0f03
I20260927 03:58:23.367802 17262 raft_consensus.cc:504] T 00000000000000000000000000000000 P b08185cf201e47eebf83a05b40d72b4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b08185cf201e47eebf83a05b40d72b4d" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 44897 } }
I20260927 03:58:23.367924 17262 leader_election.cc:302] T 00000000000000000000000000000000 P b08185cf201e47eebf83a05b40d72b4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b08185cf201e47eebf83a05b40d72b4d; no voters:
I20260927 03:58:23.368088 17262 leader_election.cc:288] T 00000000000000000000000000000000 P b08185cf201e47eebf83a05b40d72b4d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:58:23.368110 17265 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b08185cf201e47eebf83a05b40d72b4d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:58:23.368300 17265 raft_consensus.cc:686] T 00000000000000000000000000000000 P b08185cf201e47eebf83a05b40d72b4d [term 1 LEADER]: Becoming Leader. State: Replica: b08185cf201e47eebf83a05b40d72b4d, State: Running, Role: LEADER
I20260927 03:58:23.368407 17262 sys_catalog.cc:564] T 00000000000000000000000000000000 P b08185cf201e47eebf83a05b40d72b4d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:58:23.368424 17265 consensus_queue.cc:237] T 00000000000000000000000000000000 P b08185cf201e47eebf83a05b40d72b4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b08185cf201e47eebf83a05b40d72b4d" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 44897 } }
I20260927 03:58:23.368752 17269 sys_catalog.cc:455] T 00000000000000000000000000000000 P b08185cf201e47eebf83a05b40d72b4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b08185cf201e47eebf83a05b40d72b4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b08185cf201e47eebf83a05b40d72b4d" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 44897 } } }
I20260927 03:58:23.368963 17269 sys_catalog.cc:458] T 00000000000000000000000000000000 P b08185cf201e47eebf83a05b40d72b4d [sys.catalog]: This master's current role is: LEADER
I20260927 03:58:23.368736 17265 sys_catalog.cc:455] T 00000000000000000000000000000000 P b08185cf201e47eebf83a05b40d72b4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader b08185cf201e47eebf83a05b40d72b4d. Latest consensus state: current_term: 1 leader_uuid: "b08185cf201e47eebf83a05b40d72b4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b08185cf201e47eebf83a05b40d72b4d" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 44897 } } }
I20260927 03:58:23.369221 17265 sys_catalog.cc:458] T 00000000000000000000000000000000 P b08185cf201e47eebf83a05b40d72b4d [sys.catalog]: This master's current role is: LEADER
I20260927 03:58:23.369540 17276 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:58:23.370065 17276 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:58:23.371843 17276 catalog_manager.cc:1347] Generated new cluster ID: 2d6514a04e0345e198761e1da80adf14
I20260927 03:58:23.371892 17276 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 03:58:23 dist-test-slave-glgz krb5kdc[17171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 03:58:23.378364 17267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:23.368153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49210 (local address 127.15.185.126:44897)
0927 03:58:23.368399 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:23.368408 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:23.368438 (+ 30us) server_negotiation.cc:407] Connection header received
0927 03:58:23.368556 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:23.368562 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:23.368627 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:23.368834 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:23.369701 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:23.370961 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:23.371925 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:23.372346 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:23.372471 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:23.374719 (+ 2248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:23.374722 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:23.374723 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:23.374734 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:23.376885 (+ 2151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:23.377259 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:23.377262 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:23.377262 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:23.377358 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:23.377573 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:23.377574 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:23.377574 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:23.377795 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:23.377956 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:23.378040 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 03:58:23.378177 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":69,"threads_started":1}
I20260927 03:58:23.402040 17276 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:58:23.402606 17276 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:58:23.410537 17276 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b08185cf201e47eebf83a05b40d72b4d: Generated new TSK 0
I20260927 03:58:23.410766 17276 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.185.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab.
Entry for principal kudu/127.15.185.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab.
I20260927 03:58:23.439832 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.65:0
--local_ip_for_outbound_sockets=127.15.185.65
--webserver_interface=127.15.185.65
--webserver_port=0
--tserver_master_addrs=127.15.185.126:44897
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab
--principal=kudu/127.15.185.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:35381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:58:23.522061 17288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:23.522259 17288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:23.522282 17288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:23.523458 17288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:23.523499 17288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:23.523520 17288 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 03:58:23.523567 17288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.65
I20260927 03:58:23.525033 17288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:35381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab
--principal=kudu/127.15.185.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.185.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.185.126:44897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.65
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:23.525251 17288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:23.525465 17288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:23.530606 17294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:23.530652 17288 server_base.cc:1034] running on GCE node
W20260927 03:58:23.530604 17296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:23.530745 17293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:23.531148 17288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:23.531397 17288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:23.532572 17288 hybrid_clock.cc:648] HybridClock initialized: now 1790481503532567 us; error 37 us; skew 500 ppm
Sep 27 03:58:23 dist-test-slave-glgz krb5kdc[17171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481503, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:23.534982 17288 init.cc:377] Logged in from keytab as kudu/127.15.185.65@KRBTEST.COM (short username kudu)
I20260927 03:58:23.535439 17288 webserver.cc:460] Webserver started at http://127.15.185.65:42453/ using document root <none> and password file <none>
I20260927 03:58:23.535676 17288 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:23.535744 17288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:23.535876 17288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:23.536924 17288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/ts-0/data/instance:
uuid: "a07bba424c05479eb82d96d1852e4210"
format_stamp: "Formatted at 2026-09-27 03:58:23 on dist-test-slave-glgz"
server_key: "99f39a00a475745e56247e366b97f362"
server_key_iv: "fdce5d692b853518bddfdafaeb6e5488"
server_key_version: "encryptionkey@0"
I20260927 03:58:23.537298 17288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/ts-0/wal/instance:
uuid: "a07bba424c05479eb82d96d1852e4210"
format_stamp: "Formatted at 2026-09-27 03:58:23 on dist-test-slave-glgz"
server_key: "99f39a00a475745e56247e366b97f362"
server_key_iv: "fdce5d692b853518bddfdafaeb6e5488"
server_key_version: "encryptionkey@0"
I20260927 03:58:23.538654 17288 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:58:23.539439 17303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:23.539762 17288 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 03:58:23.539847 17288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/ts-0/wal
uuid: "a07bba424c05479eb82d96d1852e4210"
format_stamp: "Formatted at 2026-09-27 03:58:23 on dist-test-slave-glgz"
server_key: "99f39a00a475745e56247e366b97f362"
server_key_iv: "fdce5d692b853518bddfdafaeb6e5488"
server_key_version: "encryptionkey@0"
I20260927 03:58:23.539901 17288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:23.546586 17288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:23.548390 17288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:23.548496 17288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:23.548910 17288 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:58:23.548980 17288 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:23.549023 17288 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:58:23.549052 17288 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:23.555528 17288 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.65:40241
I20260927 03:58:23.555598 17418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.65:40241 every 8 connection(s)
I20260927 03:58:23.555897 17288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
Sep 27 03:58:23 dist-test-slave-glgz krb5kdc[17171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481503, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.65@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 03:58:23.558779 17267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:23.549370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:54555 (local address 127.15.185.126:44897)
0927 03:58:23.549413 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:23.549416 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:23.549627 (+ 211us) server_negotiation.cc:407] Connection header received
0927 03:58:23.550136 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:23.550137 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:23.550150 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:23.550209 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:23.551062 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:23.551903 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:23.552926 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:23.553284 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:23.553373 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:23.556481 (+ 3108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:23.556482 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:23.556483 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:23.556489 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:23.557674 (+ 1185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:23.558023 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:23.558024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:23.558025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:23.558073 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:23.558290 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:23.558290 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:23.558291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:23.558387 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:23.558437 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:23.558655 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 03:58:23.558716 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 03:58:23.558990 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:23.549362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:44897 (local address 127.15.185.65:54555)
0927 03:58:23.549516 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:23.549549 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:23.550033 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:23.550274 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:23.550278 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:23.550658 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:23.550944 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:23.550950 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:23.552019 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:23.552021 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:23.552852 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:23.552854 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:23.553389 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:23.553390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:23.553537 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:23.554615 (+ 1078us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:23.554624 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:23.556387 (+ 1763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:23.557743 (+ 1356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:23.557748 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:23.557750 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:23.557958 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:23.558130 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:23.558131 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:23.558131 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:23.558230 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:23.558440 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:23.558444 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:23.558549 (+ 105us) client_negotiation.cc:769] Sending connection context
0927 03:58:23.558635 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 03:58:23.558748 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":86,"threads_started":1}
I20260927 03:58:23.559473 17419 heartbeater.cc:347] Connected to a master server at 127.15.185.126:44897
I20260927 03:58:23.559573 17419 heartbeater.cc:464] Registering TS with master...
I20260927 03:58:23.559769 17419 heartbeater.cc:511] Master 127.15.185.126:44897 requested a full tablet report, sending...
I20260927 03:58:23.560319 17215 ts_manager.cc:194] Registered new tserver with Master: a07bba424c05479eb82d96d1852e4210 (127.15.185.65:40241)
I20260927 03:58:23.560945 17215 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.185.65@KRBTEST.COM'} at 127.15.185.65:54555
I20260927 03:58:23.564530 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 17288
I20260927 03:58:23.564641 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/ts-0/wal/instance
I20260927 03:58:23.564847 16101 external_mini_cluster.cc:1428] Setting key b3d9b02a8e5f5e747c0e541c41bdd948
I20260927 03:58:23.566408 17267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:23.561162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:33843 (local address 127.15.185.126:44897)
0927 03:58:23.561328 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:23.561329 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:23.561335 (+ 6us) server_negotiation.cc:407] Connection header received
0927 03:58:23.561350 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:23.561351 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:23.561359 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:23.561424 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:23.561744 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:23.562201 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:23.562898 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:23.563164 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:23.563245 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:23.564073 (+ 828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:23.564074 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:23.564075 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:23.564081 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:23.565362 (+ 1281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:23.565746 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:23.565747 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:23.565747 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:23.565801 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:23.566003 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:23.566004 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:23.566004 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:23.566094 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:23.566144 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:23.566293 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 03:58:23.566352 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147}
I20260927 03:58:23.566534 17425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:23.560450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:44897 (local address 127.15.185.65:33843)
0927 03:58:23.560562 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:23.560572 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:23.560624 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:23.561442 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:23.561443 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:23.561616 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:23.561663 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:23.561665 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:23.562297 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:23.562298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:23.562804 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:23.562806 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:23.563245 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:23.563246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:23.563325 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:23.563676 (+ 351us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:23.563680 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:23.563932 (+ 252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:23.565462 (+ 1530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:23.565463 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:23.565464 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:23.565667 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:23.565863 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:23.565864 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:23.565864 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:23.565932 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:23.566153 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:23.566155 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:23.566203 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 03:58:23.566278 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 03:58:23.566391 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":59,"threads_started":1}
WARNING: no policy specified for kudu/127.15.185.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab.
Entry for principal kudu/127.15.185.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab.
I20260927 03:58:23.589850 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.66:0
--local_ip_for_outbound_sockets=127.15.185.66
--webserver_interface=127.15.185.66
--webserver_port=0
--tserver_master_addrs=127.15.185.126:44897
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab
--principal=kudu/127.15.185.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:35381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:58:23.670269 17428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:23.670507 17428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:23.670547 17428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:23.671801 17428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:23.671869 17428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:23.671906 17428 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 03:58:23.671964 17428 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.66
I20260927 03:58:23.673420 17428 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:35381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab
--principal=kudu/127.15.185.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.185.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.185.126:44897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.66
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:23.673696 17428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:23.673944 17428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:23.679157 17433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:23.679235 17428 server_base.cc:1034] running on GCE node
W20260927 03:58:23.679165 17434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:23.679368 17436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:23.679613 17428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:23.679874 17428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:23.681039 17428 hybrid_clock.cc:648] HybridClock initialized: now 1790481503681016 us; error 56 us; skew 500 ppm
Sep 27 03:58:23 dist-test-slave-glgz krb5kdc[17171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481503, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:23.683758 17428 init.cc:377] Logged in from keytab as kudu/127.15.185.66@KRBTEST.COM (short username kudu)
I20260927 03:58:23.684279 17428 webserver.cc:460] Webserver started at http://127.15.185.66:41511/ using document root <none> and password file <none>
I20260927 03:58:23.684514 17428 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:23.684583 17428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:23.684736 17428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:23.685802 17428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/ts-1/data/instance:
uuid: "b8beb7b111314835aee34dc4c947a183"
format_stamp: "Formatted at 2026-09-27 03:58:23 on dist-test-slave-glgz"
server_key: "12bdee62b5a10326e18a023c45c1bd82"
server_key_iv: "1651e62cbe328d87be1e523f113f5565"
server_key_version: "encryptionkey@0"
I20260927 03:58:23.686187 17428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/ts-1/wal/instance:
uuid: "b8beb7b111314835aee34dc4c947a183"
format_stamp: "Formatted at 2026-09-27 03:58:23 on dist-test-slave-glgz"
server_key: "12bdee62b5a10326e18a023c45c1bd82"
server_key_iv: "1651e62cbe328d87be1e523f113f5565"
server_key_version: "encryptionkey@0"
I20260927 03:58:23.687687 17428 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:58:23.688553 17443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:23.688755 17428 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:58:23.688854 17428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/ts-1/wal
uuid: "b8beb7b111314835aee34dc4c947a183"
format_stamp: "Formatted at 2026-09-27 03:58:23 on dist-test-slave-glgz"
server_key: "12bdee62b5a10326e18a023c45c1bd82"
server_key_iv: "1651e62cbe328d87be1e523f113f5565"
server_key_version: "encryptionkey@0"
I20260927 03:58:23.688939 17428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:23.698796 17428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:23.701128 17428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:23.701296 17428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:23.701804 17428 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:58:23.701865 17428 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:23.701910 17428 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:58:23.701962 17428 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:23.708987 17428 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.66:39785
I20260927 03:58:23.709035 17558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.66:39785 every 8 connection(s)
Sep 27 03:58:23 dist-test-slave-glgz krb5kdc[17171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481503, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.66@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 03:58:23.709374 17428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
I20260927 03:58:23.712401 17267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:23.702486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:35583 (local address 127.15.185.126:44897)
0927 03:58:23.702543 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:23.702545 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:23.702741 (+ 196us) server_negotiation.cc:407] Connection header received
0927 03:58:23.703363 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:23.703365 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:23.703380 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:23.703458 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:23.704479 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:23.705328 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:23.706935 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:23.707279 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:23.707368 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:23.709885 (+ 2517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:23.709887 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:23.709887 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:23.709896 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:23.711225 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:23.711577 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:23.711578 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:23.711578 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:23.711629 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:23.711819 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:23.711819 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:23.711820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:23.711983 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:23.712037 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:23.712266 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 03:58:23.712329 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 03:58:23.712615 17452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:23.702478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:44897 (local address 127.15.185.66:35583)
0927 03:58:23.702626 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:23.702661 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:23.703267 (+ 606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:23.703573 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:23.703577 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:23.704041 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:23.704379 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:23.704387 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:23.705433 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:23.705434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:23.706829 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:23.706833 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:23.707374 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:23.707375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:23.707514 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:23.708124 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:23.708132 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:23.709794 (+ 1662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:23.711294 (+ 1500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:23.711299 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:23.711301 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:23.711514 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:23.711671 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:23.711671 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:23.711672 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:23.711772 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:23.712043 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:23.712048 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:23.712166 (+ 118us) client_negotiation.cc:769] Sending connection context
0927 03:58:23.712248 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 03:58:23.712367 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":75,"threads_started":1}
I20260927 03:58:23.713112 17559 heartbeater.cc:347] Connected to a master server at 127.15.185.126:44897
I20260927 03:58:23.713198 17559 heartbeater.cc:464] Registering TS with master...
I20260927 03:58:23.713397 17559 heartbeater.cc:511] Master 127.15.185.126:44897 requested a full tablet report, sending...
I20260927 03:58:23.713825 17216 ts_manager.cc:194] Registered new tserver with Master: b8beb7b111314835aee34dc4c947a183 (127.15.185.66:39785)
I20260927 03:58:23.714332 17216 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.185.66@KRBTEST.COM'} at 127.15.185.66:35583
I20260927 03:58:23.714617 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 17428
I20260927 03:58:23.714725 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/ts-1/wal/instance
I20260927 03:58:23.714893 16101 external_mini_cluster.cc:1428] Setting key 3897c4489f8b290ccba028166feb97a8
I20260927 03:58:23.720199 17565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:23.714120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:44897 (local address 127.15.185.66:56593)
0927 03:58:23.714247 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:23.714257 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:23.714317 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:23.714524 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:23.714526 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:23.714735 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:23.714783 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:23.714785 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:23.715629 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:23.715630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:23.716135 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:23.716136 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:23.716523 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:23.716523 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:23.716598 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:23.717078 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:23.717083 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:23.717373 (+ 290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:23.719137 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:23.719138 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:23.719138 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:23.719308 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:23.719525 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:23.719526 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:23.719526 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:23.719576 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:23.719938 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:23.719938 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:23.719973 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 03:58:23.720023 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 03:58:23.720122 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":77,"threads_started":1}
I20260927 03:58:23.720381 17267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:23.714205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:56593 (local address 127.15.185.126:44897)
0927 03:58:23.714242 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:23.714244 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:23.714309 (+ 65us) server_negotiation.cc:407] Connection header received
0927 03:58:23.714374 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:23.714375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:23.714382 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:23.714451 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:23.714873 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:23.715545 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:23.716237 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:23.716440 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:23.716522 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:23.717482 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:23.717483 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:23.717483 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:23.717490 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:23.719016 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:23.719389 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:23.719390 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:23.719391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:23.719455 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:23.719641 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:23.719642 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:23.719642 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:23.719865 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:23.719947 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:23.720062 (+ 115us) server_negotiation.cc:299] Negotiation successful
0927 03:58:23.720144 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
WARNING: no policy specified for kudu/127.15.185.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab.
Entry for principal kudu/127.15.185.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab.
I20260927 03:58:23.741973 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.67:0
--local_ip_for_outbound_sockets=127.15.185.67
--webserver_interface=127.15.185.67
--webserver_port=0
--tserver_master_addrs=127.15.185.126:44897
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab
--principal=kudu/127.15.185.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:35381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:58:23.823184 17568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:23.823400 17568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:23.823426 17568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:23.824575 17568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:23.824615 17568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:23.824638 17568 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 03:58:23.824656 17568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.67
I20260927 03:58:23.826000 17568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:35381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab
--principal=kudu/127.15.185.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.185.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.185.126:44897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.67
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:23.826238 17568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:23.826447 17568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:23.831338 17576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:23.831338 17574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:23.831405 17573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:23.831765 17568 server_base.cc:1034] running on GCE node
I20260927 03:58:23.832000 17568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:23.832265 17568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:23.833436 17568 hybrid_clock.cc:648] HybridClock initialized: now 1790481503833411 us; error 38 us; skew 500 ppm
Sep 27 03:58:23 dist-test-slave-glgz krb5kdc[17171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481503, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:23.836162 17568 init.cc:377] Logged in from keytab as kudu/127.15.185.67@KRBTEST.COM (short username kudu)
I20260927 03:58:23.836690 17568 webserver.cc:460] Webserver started at http://127.15.185.67:45979/ using document root <none> and password file <none>
I20260927 03:58:23.836947 17568 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:23.837019 17568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:23.837172 17568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:23.838217 17568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/ts-2/data/instance:
uuid: "ec4a483880344b1c9602337e8de06098"
format_stamp: "Formatted at 2026-09-27 03:58:23 on dist-test-slave-glgz"
server_key: "cf4d7d15ef40f1b56999e04957e616b7"
server_key_iv: "d67dc8bfb63664729360fbf903d44d8d"
server_key_version: "encryptionkey@0"
I20260927 03:58:23.838586 17568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/ts-2/wal/instance:
uuid: "ec4a483880344b1c9602337e8de06098"
format_stamp: "Formatted at 2026-09-27 03:58:23 on dist-test-slave-glgz"
server_key: "cf4d7d15ef40f1b56999e04957e616b7"
server_key_iv: "d67dc8bfb63664729360fbf903d44d8d"
server_key_version: "encryptionkey@0"
I20260927 03:58:23.840022 17568 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260927 03:58:23.840829 17583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:23.841017 17568 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:58:23.841121 17568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/ts-2/wal
uuid: "ec4a483880344b1c9602337e8de06098"
format_stamp: "Formatted at 2026-09-27 03:58:23 on dist-test-slave-glgz"
server_key: "cf4d7d15ef40f1b56999e04957e616b7"
server_key_iv: "d67dc8bfb63664729360fbf903d44d8d"
server_key_version: "encryptionkey@0"
I20260927 03:58:23.841194 17568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:23.852008 17568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:23.854242 17568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:23.854401 17568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:23.854936 17568 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:58:23.854995 17568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:23.855034 17568 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:58:23.855124 17568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 03:58:23 dist-test-slave-glgz krb5kdc[17171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481503, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.67@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 03:58:23.862380 17568 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.67:42711
I20260927 03:58:23.862473 17698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.67:42711 every 8 connection(s)
I20260927 03:58:23.862735 17568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
I20260927 03:58:23.864971 17267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:23.855296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:33929 (local address 127.15.185.126:44897)
0927 03:58:23.855343 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:23.855346 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:23.855613 (+ 267us) server_negotiation.cc:407] Connection header received
0927 03:58:23.856064 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:23.856065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:23.856080 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:23.856160 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:23.856816 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:23.857308 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:23.858364 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:23.858598 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:23.858688 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:23.861825 (+ 3137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:23.861827 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:23.861827 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:23.861835 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:23.863520 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:23.863949 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:23.863950 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:23.863950 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:23.864007 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:23.864302 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:23.864303 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:23.864303 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:23.864425 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:23.864504 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:23.864800 (+ 296us) server_negotiation.cc:299] Negotiation successful
0927 03:58:23.864879 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 03:58:23.865180 17592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:23.855308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:44897 (local address 127.15.185.67:33929)
0927 03:58:23.855471 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:23.855510 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:23.855970 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:23.856232 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:23.856235 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:23.856453 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:23.856718 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:23.856723 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:23.857403 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:23.857404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:23.858269 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:23.858271 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:23.858698 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:23.858699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:23.858862 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:23.859412 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:23.859424 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:23.861715 (+ 2291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:23.863626 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:23.863632 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:23.863634 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:23.863853 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:23.864089 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:23.864090 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:23.864091 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:23.864243 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:23.864494 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:23.864500 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:23.864651 (+ 151us) client_negotiation.cc:769] Sending connection context
0927 03:58:23.864775 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 03:58:23.864933 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":98,"threads_started":1}
I20260927 03:58:23.865614 17699 heartbeater.cc:347] Connected to a master server at 127.15.185.126:44897
I20260927 03:58:23.865697 17699 heartbeater.cc:464] Registering TS with master...
I20260927 03:58:23.865875 17699 heartbeater.cc:511] Master 127.15.185.126:44897 requested a full tablet report, sending...
I20260927 03:58:23.866254 17215 ts_manager.cc:194] Registered new tserver with Master: ec4a483880344b1c9602337e8de06098 (127.15.185.67:42711)
I20260927 03:58:23.866739 17215 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.185.67@KRBTEST.COM'} at 127.15.185.67:33929
I20260927 03:58:23.867090 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 17568
I20260927 03:58:23.867187 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/ts-2/wal/instance
I20260927 03:58:23.867352 16101 external_mini_cluster.cc:1428] Setting key e567573fc56adb9f43b3ca637dcc3c9d
I20260927 03:58:23.868294 16101 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:58:23.872818 17267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:23.866692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:50395 (local address 127.15.185.126:44897)
0927 03:58:23.866732 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:23.866734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:23.866852 (+ 118us) server_negotiation.cc:407] Connection header received
0927 03:58:23.866955 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:23.866956 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:23.866966 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:23.867028 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:23.867403 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:23.868105 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:23.868973 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:23.869242 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:23.869288 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:23.870292 (+ 1004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:23.870293 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:23.870293 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:23.870299 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:23.871752 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:23.872147 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:23.872147 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:23.872147 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:23.872193 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:23.872397 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:23.872398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:23.872398 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:23.872496 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:23.872543 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:23.872710 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 03:58:23.872762 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 03:58:23.872907 17705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:23.866624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:44897 (local address 127.15.185.67:50395)
0927 03:58:23.866770 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:23.866804 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:23.866875 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:23.867034 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:23.867035 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:23.867250 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:23.867317 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:23.867319 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:23.868200 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:23.868201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:23.868859 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:23.868861 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:23.869379 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:23.869380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:23.869452 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:23.869884 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:23.869889 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:23.870219 (+ 330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:23.871835 (+ 1616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:23.871836 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:23.871836 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:23.872062 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:23.872259 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:23.872260 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:23.872260 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:23.872332 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:23.872558 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:23.872559 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:23.872609 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 03:58:23.872685 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 03:58:23.872792 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":64,"threads_started":1}
Sep 27 03:58:23 dist-test-slave-glgz krb5kdc[17171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481503, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 03:58:23 dist-test-slave-glgz krb5kdc[17171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481503, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 03:58:23.891000 17267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:23.884335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49222 (local address 127.15.185.126:44897)
0927 03:58:23.884381 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:23.884383 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:23.884390 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:58:23.884455 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:23.884456 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:23.884464 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:23.884514 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:23.885022 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:23.885490 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:23.886158 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:23.886329 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:23.886377 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:23.888543 (+ 2166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:23.888544 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:23.888544 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:23.888551 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:23.889956 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:23.890296 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:23.890297 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:23.890297 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:23.890351 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:23.890526 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:23.890527 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:23.890527 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:23.890650 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:23.890734 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:23.890811 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 03:58:23.890871 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 03:58:23.892910 17215 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49222:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:58:23.894510 17215 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:58:23.900743 17714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:23.896584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.66:39785 (local address 127.0.0.1:58534)
0927 03:58:23.896881 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:23.896923 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:23.897001 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:23.897212 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:23.897217 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:23.897219 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:58:23.897287 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:23.897288 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:23.898278 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:23.898279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:23.899462 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:23.899464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:23.900401 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:23.900402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:23.900499 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:23.900513 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 03:58:23.900596 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 03:58:23.900654 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":2,"thread_start_us":59,"threads_started":1}
I20260927 03:58:23.900777 17715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:23.896681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58534 (local address 127.15.185.66:39785)
0927 03:58:23.896924 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:23.896929 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:23.896976 (+ 47us) server_negotiation.cc:407] Connection header received
0927 03:58:23.897090 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:23.897092 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:23.897110 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:23.897209 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:58:23.897346 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:23.898177 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:23.899557 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:23.900294 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:23.900366 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:23.900430 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:23.900619 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 03:58:23.900660 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":106,"threads_started":1}
I20260927 03:58:23.900866 17717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:23.896750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53228 (local address 127.15.185.67:42711)
0927 03:58:23.896924 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:23.896929 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:23.897168 (+ 239us) server_negotiation.cc:407] Connection header received
0927 03:58:23.897295 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:23.897297 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:23.897313 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:23.897369 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:58:23.897591 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:23.898680 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:23.899882 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:23.900411 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:23.900482 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:23.900539 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:23.900717 (+ 178us) server_negotiation.cc:299] Negotiation successful
0927 03:58:23.900760 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":102,"threads_started":1}
I20260927 03:58:23.901156 17718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:23.896964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39280 (local address 127.15.185.65:40241)
0927 03:58:23.897125 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:23.897129 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:23.897258 (+ 129us) server_negotiation.cc:407] Connection header received
0927 03:58:23.897295 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:23.897297 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:23.897313 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:23.897383 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:58:23.897571 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:23.898680 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:23.899975 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:23.900710 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:23.900778 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:23.900827 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:23.901026 (+ 199us) server_negotiation.cc:299] Negotiation successful
0927 03:58:23.901064 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":83,"threads_started":1}
I20260927 03:58:23.901160 17716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:23.896783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.65:40241 (local address 127.0.0.1:39280)
0927 03:58:23.897149 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:23.897158 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:23.897202 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:23.897441 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:23.897442 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:23.897445 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:58:23.897499 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:23.897500 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:23.898777 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:23.898777 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:23.899876 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:23.899878 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:23.900864 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:23.900865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:23.900939 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:23.900953 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 03:58:23.901011 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 03:58:23.901052 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":52,"thread_start_us":37,"threads_started":1}
I20260927 03:58:23.901062 17713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:23.896588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.67:42711 (local address 127.0.0.1:53228)
0927 03:58:23.897104 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:23.897112 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:23.897197 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:23.897451 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:23.897452 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:23.897453 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:58:23.897497 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:23.897498 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:23.898786 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:23.898787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:23.899798 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:23.899800 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:23.900571 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:23.900572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:23.900626 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:23.900629 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:58:23.900970 (+ 341us) client_negotiation.cc:240] Negotiation successful
0927 03:58:23.901007 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"mutex_wait_us":4,"thread_start_us":55,"threads_started":1}
I20260927 03:58:23.901901 17353 tablet_service.cc:1467] Processing CreateTablet for tablet 521cf6944be24b8b93ce28f1dc55b619 (DEFAULT_TABLE table=test-table [id=108d27f6dd1f4f12a330d8cf5468ff03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:58:23.901891 17633 tablet_service.cc:1467] Processing CreateTablet for tablet 521cf6944be24b8b93ce28f1dc55b619 (DEFAULT_TABLE table=test-table [id=108d27f6dd1f4f12a330d8cf5468ff03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:58:23.902257 17633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 521cf6944be24b8b93ce28f1dc55b619. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:23.902257 17353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 521cf6944be24b8b93ce28f1dc55b619. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:23.902789 17493 tablet_service.cc:1467] Processing CreateTablet for tablet 521cf6944be24b8b93ce28f1dc55b619 (DEFAULT_TABLE table=test-table [id=108d27f6dd1f4f12a330d8cf5468ff03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:58:23.903151 17493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 521cf6944be24b8b93ce28f1dc55b619. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:23.905354 17719 tablet_bootstrap.cc:492] T 521cf6944be24b8b93ce28f1dc55b619 P a07bba424c05479eb82d96d1852e4210: Bootstrap starting.
I20260927 03:58:23.905965 17719 tablet_bootstrap.cc:654] T 521cf6944be24b8b93ce28f1dc55b619 P a07bba424c05479eb82d96d1852e4210: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:23.906276 17719 log.cc:824] T 521cf6944be24b8b93ce28f1dc55b619 P a07bba424c05479eb82d96d1852e4210: Log is configured to *not* fsync() on all Append() calls
I20260927 03:58:23.906507 17721 tablet_bootstrap.cc:492] T 521cf6944be24b8b93ce28f1dc55b619 P b8beb7b111314835aee34dc4c947a183: Bootstrap starting.
I20260927 03:58:23.906606 17720 tablet_bootstrap.cc:492] T 521cf6944be24b8b93ce28f1dc55b619 P ec4a483880344b1c9602337e8de06098: Bootstrap starting.
I20260927 03:58:23.907192 17719 tablet_bootstrap.cc:492] T 521cf6944be24b8b93ce28f1dc55b619 P a07bba424c05479eb82d96d1852e4210: No bootstrap required, opened a new log
I20260927 03:58:23.907272 17721 tablet_bootstrap.cc:654] T 521cf6944be24b8b93ce28f1dc55b619 P b8beb7b111314835aee34dc4c947a183: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:23.907274 17719 ts_tablet_manager.cc:1397] T 521cf6944be24b8b93ce28f1dc55b619 P a07bba424c05479eb82d96d1852e4210: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:58:23.907510 17720 tablet_bootstrap.cc:654] T 521cf6944be24b8b93ce28f1dc55b619 P ec4a483880344b1c9602337e8de06098: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:23.907603 17721 log.cc:824] T 521cf6944be24b8b93ce28f1dc55b619 P b8beb7b111314835aee34dc4c947a183: Log is configured to *not* fsync() on all Append() calls
I20260927 03:58:23.907827 17720 log.cc:824] T 521cf6944be24b8b93ce28f1dc55b619 P ec4a483880344b1c9602337e8de06098: Log is configured to *not* fsync() on all Append() calls
I20260927 03:58:23.908494 17720 tablet_bootstrap.cc:492] T 521cf6944be24b8b93ce28f1dc55b619 P ec4a483880344b1c9602337e8de06098: No bootstrap required, opened a new log
I20260927 03:58:23.908490 17721 tablet_bootstrap.cc:492] T 521cf6944be24b8b93ce28f1dc55b619 P b8beb7b111314835aee34dc4c947a183: No bootstrap required, opened a new log
I20260927 03:58:23.908563 17720 ts_tablet_manager.cc:1397] T 521cf6944be24b8b93ce28f1dc55b619 P ec4a483880344b1c9602337e8de06098: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260927 03:58:23.908560 17721 ts_tablet_manager.cc:1397] T 521cf6944be24b8b93ce28f1dc55b619 P b8beb7b111314835aee34dc4c947a183: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:58:23.909114 17719 raft_consensus.cc:348] T 521cf6944be24b8b93ce28f1dc55b619 P a07bba424c05479eb82d96d1852e4210 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec4a483880344b1c9602337e8de06098" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 42711 } } peers { permanent_uuid: "a07bba424c05479eb82d96d1852e4210" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 40241 } } peers { permanent_uuid: "b8beb7b111314835aee34dc4c947a183" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 39785 } }
I20260927 03:58:23.909271 17719 raft_consensus.cc:374] T 521cf6944be24b8b93ce28f1dc55b619 P a07bba424c05479eb82d96d1852e4210 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:23.909315 17719 raft_consensus.cc:729] T 521cf6944be24b8b93ce28f1dc55b619 P a07bba424c05479eb82d96d1852e4210 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a07bba424c05479eb82d96d1852e4210, State: Initialized, Role: FOLLOWER
I20260927 03:58:23.909421 17719 consensus_queue.cc:260] T 521cf6944be24b8b93ce28f1dc55b619 P a07bba424c05479eb82d96d1852e4210 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec4a483880344b1c9602337e8de06098" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 42711 } } peers { permanent_uuid: "a07bba424c05479eb82d96d1852e4210" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 40241 } } peers { permanent_uuid: "b8beb7b111314835aee34dc4c947a183" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 39785 } }
I20260927 03:58:23.909693 17719 ts_tablet_manager.cc:1428] T 521cf6944be24b8b93ce28f1dc55b619 P a07bba424c05479eb82d96d1852e4210: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 03:58:23.909696 17419 heartbeater.cc:503] Master 127.15.185.126:44897 was elected leader, sending a full tablet report...
I20260927 03:58:23.910565 17721 raft_consensus.cc:348] T 521cf6944be24b8b93ce28f1dc55b619 P b8beb7b111314835aee34dc4c947a183 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec4a483880344b1c9602337e8de06098" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 42711 } } peers { permanent_uuid: "a07bba424c05479eb82d96d1852e4210" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 40241 } } peers { permanent_uuid: "b8beb7b111314835aee34dc4c947a183" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 39785 } }
I20260927 03:58:23.910561 17720 raft_consensus.cc:348] T 521cf6944be24b8b93ce28f1dc55b619 P ec4a483880344b1c9602337e8de06098 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec4a483880344b1c9602337e8de06098" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 42711 } } peers { permanent_uuid: "a07bba424c05479eb82d96d1852e4210" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 40241 } } peers { permanent_uuid: "b8beb7b111314835aee34dc4c947a183" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 39785 } }
I20260927 03:58:23.910717 17721 raft_consensus.cc:374] T 521cf6944be24b8b93ce28f1dc55b619 P b8beb7b111314835aee34dc4c947a183 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:23.910717 17720 raft_consensus.cc:374] T 521cf6944be24b8b93ce28f1dc55b619 P ec4a483880344b1c9602337e8de06098 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:23.910758 17720 raft_consensus.cc:729] T 521cf6944be24b8b93ce28f1dc55b619 P ec4a483880344b1c9602337e8de06098 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec4a483880344b1c9602337e8de06098, State: Initialized, Role: FOLLOWER
I20260927 03:58:23.910758 17721 raft_consensus.cc:729] T 521cf6944be24b8b93ce28f1dc55b619 P b8beb7b111314835aee34dc4c947a183 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8beb7b111314835aee34dc4c947a183, State: Initialized, Role: FOLLOWER
I20260927 03:58:23.910861 17720 consensus_queue.cc:260] T 521cf6944be24b8b93ce28f1dc55b619 P ec4a483880344b1c9602337e8de06098 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec4a483880344b1c9602337e8de06098" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 42711 } } peers { permanent_uuid: "a07bba424c05479eb82d96d1852e4210" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 40241 } } peers { permanent_uuid: "b8beb7b111314835aee34dc4c947a183" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 39785 } }
I20260927 03:58:23.910879 17721 consensus_queue.cc:260] T 521cf6944be24b8b93ce28f1dc55b619 P b8beb7b111314835aee34dc4c947a183 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec4a483880344b1c9602337e8de06098" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 42711 } } peers { permanent_uuid: "a07bba424c05479eb82d96d1852e4210" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 40241 } } peers { permanent_uuid: "b8beb7b111314835aee34dc4c947a183" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 39785 } }
I20260927 03:58:23.911154 17720 ts_tablet_manager.cc:1428] T 521cf6944be24b8b93ce28f1dc55b619 P ec4a483880344b1c9602337e8de06098: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 03:58:23.911154 17721 ts_tablet_manager.cc:1428] T 521cf6944be24b8b93ce28f1dc55b619 P b8beb7b111314835aee34dc4c947a183: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:58:23.911173 17559 heartbeater.cc:503] Master 127.15.185.126:44897 was elected leader, sending a full tablet report...
I20260927 03:58:23.911288 17699 heartbeater.cc:503] Master 127.15.185.126:44897 was elected leader, sending a full tablet report...
W20260927 03:58:23.960032 17560 tablet.cc:2369] T 521cf6944be24b8b93ce28f1dc55b619 P b8beb7b111314835aee34dc4c947a183: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:58:23.978915 17727 raft_consensus.cc:482] T 521cf6944be24b8b93ce28f1dc55b619 P ec4a483880344b1c9602337e8de06098 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:58:23.979123 17727 raft_consensus.cc:504] T 521cf6944be24b8b93ce28f1dc55b619 P ec4a483880344b1c9602337e8de06098 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec4a483880344b1c9602337e8de06098" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 42711 } } peers { permanent_uuid: "a07bba424c05479eb82d96d1852e4210" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 40241 } } peers { permanent_uuid: "b8beb7b111314835aee34dc4c947a183" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 39785 } }
I20260927 03:58:23.979514 17727 leader_election.cc:288] T 521cf6944be24b8b93ce28f1dc55b619 P ec4a483880344b1c9602337e8de06098 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a07bba424c05479eb82d96d1852e4210 (127.15.185.65:40241), b8beb7b111314835aee34dc4c947a183 (127.15.185.66:39785)
I20260927 03:58:23.982455 17592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:23.979726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.65:40241 (local address 127.15.185.67:38687)
0927 03:58:23.979786 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:23.979804 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:23.979849 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:23.980036 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:23.980037 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:23.980038 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:58:23.980123 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:23.980125 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:23.980798 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:23.980799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:23.981613 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:23.981616 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:23.982171 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:23.982172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:23.982265 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:23.982270 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:58:23.982317 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 03:58:23.982368 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 03:58:23.982460 17718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:23.979810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:38687 (local address 127.15.185.65:40241)
0927 03:58:23.979850 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:23.979852 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:23.979905 (+ 53us) server_negotiation.cc:407] Connection header received
0927 03:58:23.979938 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:23.979940 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:23.979951 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:23.980005 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:58:23.980203 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:23.980711 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:23.981716 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:23.982085 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:23.982139 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:23.982146 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:23.982346 (+ 200us) server_negotiation.cc:299] Negotiation successful
0927 03:58:23.982388 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 03:58:23.982503 17715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:23.979804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:47893 (local address 127.15.185.66:39785)
0927 03:58:23.979850 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:23.979852 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:23.980040 (+ 188us) server_negotiation.cc:407] Connection header received
0927 03:58:23.980067 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:23.980069 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:23.980078 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:23.980130 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:58:23.980324 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:23.980830 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:23.981763 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:23.982147 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:23.982220 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:23.982228 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:23.982395 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 03:58:23.982432 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 03:58:23.982563 17728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:23.979728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.66:39785 (local address 127.15.185.67:47893)
0927 03:58:23.979928 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:23.979938 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:23.979999 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:23.980142 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:23.980143 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:23.980145 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:58:23.980242 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:23.980244 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:23.980911 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:23.980912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:23.981638 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:23.981640 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:23.982227 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:23.982228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:23.982290 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:23.982293 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:58:23.982370 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 03:58:23.982406 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":13,"thread_start_us":127,"threads_started":1}
I20260927 03:58:23.982844 17372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "521cf6944be24b8b93ce28f1dc55b619" candidate_uuid: "ec4a483880344b1c9602337e8de06098" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a07bba424c05479eb82d96d1852e4210" is_pre_election: true
I20260927 03:58:23.983007 17372 raft_consensus.cc:2454] T 521cf6944be24b8b93ce28f1dc55b619 P a07bba424c05479eb82d96d1852e4210 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec4a483880344b1c9602337e8de06098 in term 0.
I20260927 03:58:23.983016 17513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "521cf6944be24b8b93ce28f1dc55b619" candidate_uuid: "ec4a483880344b1c9602337e8de06098" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8beb7b111314835aee34dc4c947a183" is_pre_election: true
I20260927 03:58:23.983182 17513 raft_consensus.cc:2454] T 521cf6944be24b8b93ce28f1dc55b619 P b8beb7b111314835aee34dc4c947a183 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec4a483880344b1c9602337e8de06098 in term 0.
I20260927 03:58:23.983297 17584 leader_election.cc:302] T 521cf6944be24b8b93ce28f1dc55b619 P ec4a483880344b1c9602337e8de06098 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a07bba424c05479eb82d96d1852e4210, ec4a483880344b1c9602337e8de06098; no voters:
I20260927 03:58:23.983431 17727 raft_consensus.cc:2781] T 521cf6944be24b8b93ce28f1dc55b619 P ec4a483880344b1c9602337e8de06098 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:58:23.983510 17727 raft_consensus.cc:482] T 521cf6944be24b8b93ce28f1dc55b619 P ec4a483880344b1c9602337e8de06098 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:58:23.983546 17727 raft_consensus.cc:3037] T 521cf6944be24b8b93ce28f1dc55b619 P ec4a483880344b1c9602337e8de06098 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:23.984341 17727 raft_consensus.cc:504] T 521cf6944be24b8b93ce28f1dc55b619 P ec4a483880344b1c9602337e8de06098 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec4a483880344b1c9602337e8de06098" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 42711 } } peers { permanent_uuid: "a07bba424c05479eb82d96d1852e4210" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 40241 } } peers { permanent_uuid: "b8beb7b111314835aee34dc4c947a183" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 39785 } }
I20260927 03:58:23.984494 17727 leader_election.cc:288] T 521cf6944be24b8b93ce28f1dc55b619 P ec4a483880344b1c9602337e8de06098 [CANDIDATE]: Term 1 election: Requested vote from peers a07bba424c05479eb82d96d1852e4210 (127.15.185.65:40241), b8beb7b111314835aee34dc4c947a183 (127.15.185.66:39785)
I20260927 03:58:23.984655 17513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "521cf6944be24b8b93ce28f1dc55b619" candidate_uuid: "ec4a483880344b1c9602337e8de06098" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8beb7b111314835aee34dc4c947a183"
I20260927 03:58:23.984755 17513 raft_consensus.cc:3037] T 521cf6944be24b8b93ce28f1dc55b619 P b8beb7b111314835aee34dc4c947a183 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:23.984757 17372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "521cf6944be24b8b93ce28f1dc55b619" candidate_uuid: "ec4a483880344b1c9602337e8de06098" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a07bba424c05479eb82d96d1852e4210"
I20260927 03:58:23.984828 17372 raft_consensus.cc:3037] T 521cf6944be24b8b93ce28f1dc55b619 P a07bba424c05479eb82d96d1852e4210 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:23.985456 17513 raft_consensus.cc:2454] T 521cf6944be24b8b93ce28f1dc55b619 P b8beb7b111314835aee34dc4c947a183 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec4a483880344b1c9602337e8de06098 in term 1.
I20260927 03:58:23.985642 17372 raft_consensus.cc:2454] T 521cf6944be24b8b93ce28f1dc55b619 P a07bba424c05479eb82d96d1852e4210 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec4a483880344b1c9602337e8de06098 in term 1.
I20260927 03:58:23.985996 17587 leader_election.cc:302] T 521cf6944be24b8b93ce28f1dc55b619 P ec4a483880344b1c9602337e8de06098 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b8beb7b111314835aee34dc4c947a183, ec4a483880344b1c9602337e8de06098; no voters:
I20260927 03:58:23.986121 17727 raft_consensus.cc:2781] T 521cf6944be24b8b93ce28f1dc55b619 P ec4a483880344b1c9602337e8de06098 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:58:23.986302 17727 raft_consensus.cc:686] T 521cf6944be24b8b93ce28f1dc55b619 P ec4a483880344b1c9602337e8de06098 [term 1 LEADER]: Becoming Leader. State: Replica: ec4a483880344b1c9602337e8de06098, State: Running, Role: LEADER
I20260927 03:58:23.986413 17727 consensus_queue.cc:237] T 521cf6944be24b8b93ce28f1dc55b619 P ec4a483880344b1c9602337e8de06098 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec4a483880344b1c9602337e8de06098" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 42711 } } peers { permanent_uuid: "a07bba424c05479eb82d96d1852e4210" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 40241 } } peers { permanent_uuid: "b8beb7b111314835aee34dc4c947a183" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 39785 } }
I20260927 03:58:23.987356 17215 catalog_manager.cc:5495] T 521cf6944be24b8b93ce28f1dc55b619 P ec4a483880344b1c9602337e8de06098 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec4a483880344b1c9602337e8de06098 (127.15.185.67). New cstate: current_term: 1 leader_uuid: "ec4a483880344b1c9602337e8de06098" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec4a483880344b1c9602337e8de06098" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 42711 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a07bba424c05479eb82d96d1852e4210" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 40241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8beb7b111314835aee34dc4c947a183" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 39785 } health_report { overall_health: UNKNOWN } } }
I20260927 03:58:23.993088 17717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:23.990723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53236 (local address 127.15.185.67:42711)
0927 03:58:23.990780 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:23.990782 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:23.990792 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:58:23.990845 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:23.990846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:23.990859 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:23.990918 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:58:23.991135 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:23.991586 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:23.992336 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:23.992509 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:23.992561 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:23.992727 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:58:23.992809 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:58:23.992919 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:23.992995 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 03:58:23.993011 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 03:58:23.994796 17372 raft_consensus.cc:1261] T 521cf6944be24b8b93ce28f1dc55b619 P a07bba424c05479eb82d96d1852e4210 [term 1 FOLLOWER]: Refusing update from remote peer ec4a483880344b1c9602337e8de06098: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:58:23.994817 17513 raft_consensus.cc:1261] T 521cf6944be24b8b93ce28f1dc55b619 P b8beb7b111314835aee34dc4c947a183 [term 1 FOLLOWER]: Refusing update from remote peer ec4a483880344b1c9602337e8de06098: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:58:23.995155 17729 consensus_queue.cc:1035] T 521cf6944be24b8b93ce28f1dc55b619 P ec4a483880344b1c9602337e8de06098 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a07bba424c05479eb82d96d1852e4210" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 40241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:58:23.995280 17727 consensus_queue.cc:1035] T 521cf6944be24b8b93ce28f1dc55b619 P ec4a483880344b1c9602337e8de06098 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8beb7b111314835aee34dc4c947a183" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 39785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:58:23.997838 17734 mvcc.cc:205] Tried to move back new op lower bound from 7333812240360333312 to 7333812240329166848. Current Snapshot: MvccSnapshot[applied={T|T < 7333812240360333312}]
I20260927 03:58:24.000811 17215 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:58:24.000891 17215 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:58:24.001703 17215 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b08185cf201e47eebf83a05b40d72b4d: Sending DeleteTablet for 3 replicas of tablet 521cf6944be24b8b93ce28f1dc55b619
I20260927 03:58:24.002049 17493 tablet_service.cc:1514] Processing DeleteTablet for tablet 521cf6944be24b8b93ce28f1dc55b619 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:58:23 UTC) from {username='kudu', principal='kudu/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:58534
I20260927 03:58:24.002128 17633 tablet_service.cc:1514] Processing DeleteTablet for tablet 521cf6944be24b8b93ce28f1dc55b619 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:58:23 UTC) from {username='kudu', principal='kudu/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:53228
I20260927 03:58:24.002131 17353 tablet_service.cc:1514] Processing DeleteTablet for tablet 521cf6944be24b8b93ce28f1dc55b619 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:58:23 UTC) from {username='kudu', principal='kudu/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:39280
I20260927 03:58:24.002369 17742 tablet_replica.cc:330] T 521cf6944be24b8b93ce28f1dc55b619 P b8beb7b111314835aee34dc4c947a183: stopping tablet replica
I20260927 03:58:24.002492 17744 tablet_replica.cc:330] T 521cf6944be24b8b93ce28f1dc55b619 P ec4a483880344b1c9602337e8de06098: stopping tablet replica
I20260927 03:58:24.002609 17744 raft_consensus.cc:2229] T 521cf6944be24b8b93ce28f1dc55b619 P ec4a483880344b1c9602337e8de06098 [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:58:24.002709 17744 raft_consensus.cc:2258] T 521cf6944be24b8b93ce28f1dc55b619 P ec4a483880344b1c9602337e8de06098 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:58:24.002660 17215 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49222:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:58:24.002827 17215 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:58:24.002858 17742 raft_consensus.cc:2229] T 521cf6944be24b8b93ce28f1dc55b619 P b8beb7b111314835aee34dc4c947a183 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:58:24.003386 17742 raft_consensus.cc:2258] T 521cf6944be24b8b93ce28f1dc55b619 P b8beb7b111314835aee34dc4c947a183 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:58:24.004037 17744 ts_tablet_manager.cc:1905] T 521cf6944be24b8b93ce28f1dc55b619 P ec4a483880344b1c9602337e8de06098: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:58:24.004174 17493 tablet_service.cc:1467] Processing CreateTablet for tablet 1afd3ce85d194a9ea6f33c3d4522987e (DEFAULT_TABLE table=test-table [id=7cb67a97bf774ffa9e37986242ba0b93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:58:24.004227 17353 tablet_service.cc:1467] Processing CreateTablet for tablet 1afd3ce85d194a9ea6f33c3d4522987e (DEFAULT_TABLE table=test-table [id=7cb67a97bf774ffa9e37986242ba0b93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:58:24.004292 17493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1afd3ce85d194a9ea6f33c3d4522987e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:24.004321 17353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1afd3ce85d194a9ea6f33c3d4522987e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:24.005671 17721 tablet_bootstrap.cc:492] T 1afd3ce85d194a9ea6f33c3d4522987e P b8beb7b111314835aee34dc4c947a183: Bootstrap starting.
I20260927 03:58:24.005715 17743 tablet_replica.cc:330] T 521cf6944be24b8b93ce28f1dc55b619 P a07bba424c05479eb82d96d1852e4210: stopping tablet replica
I20260927 03:58:24.005757 17719 tablet_bootstrap.cc:492] T 1afd3ce85d194a9ea6f33c3d4522987e P a07bba424c05479eb82d96d1852e4210: Bootstrap starting.
I20260927 03:58:24.005822 17743 raft_consensus.cc:2229] T 521cf6944be24b8b93ce28f1dc55b619 P a07bba424c05479eb82d96d1852e4210 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:58:24.005925 17743 raft_consensus.cc:2258] T 521cf6944be24b8b93ce28f1dc55b619 P a07bba424c05479eb82d96d1852e4210 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:58:24.006110 17721 tablet_bootstrap.cc:654] T 1afd3ce85d194a9ea6f33c3d4522987e P b8beb7b111314835aee34dc4c947a183: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:24.006139 17719 tablet_bootstrap.cc:654] T 1afd3ce85d194a9ea6f33c3d4522987e P a07bba424c05479eb82d96d1852e4210: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:24.006161 17744 ts_tablet_manager.cc:1918] T 521cf6944be24b8b93ce28f1dc55b619 P ec4a483880344b1c9602337e8de06098: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:58:24.006228 17744 log.cc:1196] T 521cf6944be24b8b93ce28f1dc55b619 P ec4a483880344b1c9602337e8de06098: Deleting WAL directory at /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/ts-2/wal/wals/521cf6944be24b8b93ce28f1dc55b619
I20260927 03:58:24.006330 17743 ts_tablet_manager.cc:1905] T 521cf6944be24b8b93ce28f1dc55b619 P a07bba424c05479eb82d96d1852e4210: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:58:24.006480 17744 ts_tablet_manager.cc:1939] T 521cf6944be24b8b93ce28f1dc55b619 P ec4a483880344b1c9602337e8de06098: Deleting consensus metadata
I20260927 03:58:24.006575 17633 tablet_service.cc:1467] Processing CreateTablet for tablet 1afd3ce85d194a9ea6f33c3d4522987e (DEFAULT_TABLE table=test-table [id=7cb67a97bf774ffa9e37986242ba0b93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:58:24.006691 17633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1afd3ce85d194a9ea6f33c3d4522987e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:24.006835 17742 ts_tablet_manager.cc:1905] T 521cf6944be24b8b93ce28f1dc55b619 P b8beb7b111314835aee34dc4c947a183: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:58:24.007802 17743 ts_tablet_manager.cc:1918] T 521cf6944be24b8b93ce28f1dc55b619 P a07bba424c05479eb82d96d1852e4210: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:58:24.007862 17743 log.cc:1196] T 521cf6944be24b8b93ce28f1dc55b619 P a07bba424c05479eb82d96d1852e4210: Deleting WAL directory at /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/ts-0/wal/wals/521cf6944be24b8b93ce28f1dc55b619
I20260927 03:58:24.007951 17720 tablet_bootstrap.cc:492] T 1afd3ce85d194a9ea6f33c3d4522987e P ec4a483880344b1c9602337e8de06098: Bootstrap starting.
I20260927 03:58:24.008126 17743 ts_tablet_manager.cc:1939] T 521cf6944be24b8b93ce28f1dc55b619 P a07bba424c05479eb82d96d1852e4210: Deleting consensus metadata
I20260927 03:58:24.008347 17720 tablet_bootstrap.cc:654] T 1afd3ce85d194a9ea6f33c3d4522987e P ec4a483880344b1c9602337e8de06098: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:24.008685 17200 catalog_manager.cc:4841] TS a07bba424c05479eb82d96d1852e4210 (127.15.185.65:40241): tablet 521cf6944be24b8b93ce28f1dc55b619 (table test-table [id=108d27f6dd1f4f12a330d8cf5468ff03]) successfully deleted
I20260927 03:58:24.009033 17742 ts_tablet_manager.cc:1918] T 521cf6944be24b8b93ce28f1dc55b619 P b8beb7b111314835aee34dc4c947a183: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:58:24.009100 17742 log.cc:1196] T 521cf6944be24b8b93ce28f1dc55b619 P b8beb7b111314835aee34dc4c947a183: Deleting WAL directory at /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/ts-1/wal/wals/521cf6944be24b8b93ce28f1dc55b619
I20260927 03:58:24.009153 17721 tablet_bootstrap.cc:492] T 1afd3ce85d194a9ea6f33c3d4522987e P b8beb7b111314835aee34dc4c947a183: No bootstrap required, opened a new log
I20260927 03:58:24.009193 17721 ts_tablet_manager.cc:1397] T 1afd3ce85d194a9ea6f33c3d4522987e P b8beb7b111314835aee34dc4c947a183: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260927 03:58:24.009349 17742 ts_tablet_manager.cc:1939] T 521cf6944be24b8b93ce28f1dc55b619 P b8beb7b111314835aee34dc4c947a183: Deleting consensus metadata
I20260927 03:58:24.009351 17721 raft_consensus.cc:348] T 1afd3ce85d194a9ea6f33c3d4522987e P b8beb7b111314835aee34dc4c947a183 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07bba424c05479eb82d96d1852e4210" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 40241 } } peers { permanent_uuid: "b8beb7b111314835aee34dc4c947a183" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 39785 } } peers { permanent_uuid: "ec4a483880344b1c9602337e8de06098" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 42711 } }
I20260927 03:58:24.009518 17721 raft_consensus.cc:374] T 1afd3ce85d194a9ea6f33c3d4522987e P b8beb7b111314835aee34dc4c947a183 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:24.009549 17721 raft_consensus.cc:729] T 1afd3ce85d194a9ea6f33c3d4522987e P b8beb7b111314835aee34dc4c947a183 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8beb7b111314835aee34dc4c947a183, State: Initialized, Role: FOLLOWER
I20260927 03:58:24.009593 17721 consensus_queue.cc:260] T 1afd3ce85d194a9ea6f33c3d4522987e P b8beb7b111314835aee34dc4c947a183 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07bba424c05479eb82d96d1852e4210" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 40241 } } peers { permanent_uuid: "b8beb7b111314835aee34dc4c947a183" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 39785 } } peers { permanent_uuid: "ec4a483880344b1c9602337e8de06098" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 42711 } }
I20260927 03:58:24.009684 17721 ts_tablet_manager.cc:1428] T 1afd3ce85d194a9ea6f33c3d4522987e P b8beb7b111314835aee34dc4c947a183: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260927 03:58:24.009790 17720 tablet_bootstrap.cc:492] T 1afd3ce85d194a9ea6f33c3d4522987e P ec4a483880344b1c9602337e8de06098: No bootstrap required, opened a new log
I20260927 03:58:24.009847 17720 ts_tablet_manager.cc:1397] T 1afd3ce85d194a9ea6f33c3d4522987e P ec4a483880344b1c9602337e8de06098: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 03:58:24.009835 17203 catalog_manager.cc:4841] TS b8beb7b111314835aee34dc4c947a183 (127.15.185.66:39785): tablet 521cf6944be24b8b93ce28f1dc55b619 (table test-table [id=108d27f6dd1f4f12a330d8cf5468ff03]) successfully deleted
I20260927 03:58:24.010010 17720 raft_consensus.cc:348] T 1afd3ce85d194a9ea6f33c3d4522987e P ec4a483880344b1c9602337e8de06098 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07bba424c05479eb82d96d1852e4210" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 40241 } } peers { permanent_uuid: "b8beb7b111314835aee34dc4c947a183" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 39785 } } peers { permanent_uuid: "ec4a483880344b1c9602337e8de06098" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 42711 } }
I20260927 03:58:24.010078 17720 raft_consensus.cc:374] T 1afd3ce85d194a9ea6f33c3d4522987e P ec4a483880344b1c9602337e8de06098 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:24.010108 17720 raft_consensus.cc:729] T 1afd3ce85d194a9ea6f33c3d4522987e P ec4a483880344b1c9602337e8de06098 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec4a483880344b1c9602337e8de06098, State: Initialized, Role: FOLLOWER
I20260927 03:58:24.010150 17720 consensus_queue.cc:260] T 1afd3ce85d194a9ea6f33c3d4522987e P ec4a483880344b1c9602337e8de06098 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07bba424c05479eb82d96d1852e4210" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 40241 } } peers { permanent_uuid: "b8beb7b111314835aee34dc4c947a183" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 39785 } } peers { permanent_uuid: "ec4a483880344b1c9602337e8de06098" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 42711 } }
I20260927 03:58:24.010167 17200 catalog_manager.cc:4841] TS ec4a483880344b1c9602337e8de06098 (127.15.185.67:42711): tablet 521cf6944be24b8b93ce28f1dc55b619 (table test-table [id=108d27f6dd1f4f12a330d8cf5468ff03]) successfully deleted
I20260927 03:58:24.010243 17720 ts_tablet_manager.cc:1428] T 1afd3ce85d194a9ea6f33c3d4522987e P ec4a483880344b1c9602337e8de06098: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:24.010458 17719 tablet_bootstrap.cc:492] T 1afd3ce85d194a9ea6f33c3d4522987e P a07bba424c05479eb82d96d1852e4210: No bootstrap required, opened a new log
I20260927 03:58:24.010505 17719 ts_tablet_manager.cc:1397] T 1afd3ce85d194a9ea6f33c3d4522987e P a07bba424c05479eb82d96d1852e4210: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260927 03:58:24.010661 17719 raft_consensus.cc:348] T 1afd3ce85d194a9ea6f33c3d4522987e P a07bba424c05479eb82d96d1852e4210 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07bba424c05479eb82d96d1852e4210" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 40241 } } peers { permanent_uuid: "b8beb7b111314835aee34dc4c947a183" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 39785 } } peers { permanent_uuid: "ec4a483880344b1c9602337e8de06098" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 42711 } }
I20260927 03:58:24.010736 17719 raft_consensus.cc:374] T 1afd3ce85d194a9ea6f33c3d4522987e P a07bba424c05479eb82d96d1852e4210 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:24.010761 17719 raft_consensus.cc:729] T 1afd3ce85d194a9ea6f33c3d4522987e P a07bba424c05479eb82d96d1852e4210 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a07bba424c05479eb82d96d1852e4210, State: Initialized, Role: FOLLOWER
I20260927 03:58:24.010810 17719 consensus_queue.cc:260] T 1afd3ce85d194a9ea6f33c3d4522987e P a07bba424c05479eb82d96d1852e4210 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07bba424c05479eb82d96d1852e4210" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 40241 } } peers { permanent_uuid: "b8beb7b111314835aee34dc4c947a183" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 39785 } } peers { permanent_uuid: "ec4a483880344b1c9602337e8de06098" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 42711 } }
I20260927 03:58:24.010905 17719 ts_tablet_manager.cc:1428] T 1afd3ce85d194a9ea6f33c3d4522987e P a07bba424c05479eb82d96d1852e4210: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260927 03:58:24.056632 17420 tablet.cc:2369] T 1afd3ce85d194a9ea6f33c3d4522987e P a07bba424c05479eb82d96d1852e4210: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 03:58:24.113343 17700 tablet.cc:2369] T 1afd3ce85d194a9ea6f33c3d4522987e P ec4a483880344b1c9602337e8de06098: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:58:24.323027 17725 raft_consensus.cc:482] T 1afd3ce85d194a9ea6f33c3d4522987e P a07bba424c05479eb82d96d1852e4210 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:58:24.323225 17725 raft_consensus.cc:504] T 1afd3ce85d194a9ea6f33c3d4522987e P a07bba424c05479eb82d96d1852e4210 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07bba424c05479eb82d96d1852e4210" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 40241 } } peers { permanent_uuid: "b8beb7b111314835aee34dc4c947a183" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 39785 } } peers { permanent_uuid: "ec4a483880344b1c9602337e8de06098" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 42711 } }
I20260927 03:58:24.323573 17725 leader_election.cc:288] T 1afd3ce85d194a9ea6f33c3d4522987e P a07bba424c05479eb82d96d1852e4210 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8beb7b111314835aee34dc4c947a183 (127.15.185.66:39785), ec4a483880344b1c9602337e8de06098 (127.15.185.67:42711)
I20260927 03:58:24.326480 17717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:24.323924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:59753 (local address 127.15.185.67:42711)
0927 03:58:24.323976 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:24.323979 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:24.324059 (+ 80us) server_negotiation.cc:407] Connection header received
0927 03:58:24.324122 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:24.324123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:24.324134 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:24.324190 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:58:24.324373 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:24.324813 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:24.325754 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:24.326131 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:24.326220 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:24.326229 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:24.326380 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 03:58:24.326413 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 03:58:24.326506 17748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:24.323775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.66:39785 (local address 127.15.185.65:60811)
0927 03:58:24.323898 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:24.323910 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:24.323975 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:24.324129 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:24.324130 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:24.324133 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:58:24.324212 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:24.324214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:24.324901 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:24.324902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:24.325676 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:24.325678 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:24.326224 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:24.326225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:24.326296 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:24.326299 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:58:24.326360 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 03:58:24.326406 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":52,"thread_start_us":70,"threads_started":1}
I20260927 03:58:24.326499 17715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:24.323866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:60811 (local address 127.15.185.66:39785)
0927 03:58:24.323955 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:24.323958 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:24.323970 (+ 12us) server_negotiation.cc:407] Connection header received
0927 03:58:24.324045 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:24.324046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:24.324056 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:24.324113 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:58:24.324302 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:24.324809 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:24.325797 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:24.326133 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:24.326226 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:24.326233 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:24.326390 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 03:58:24.326434 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 03:58:24.326509 17749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:24.323819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.67:42711 (local address 127.15.185.65:59753)
0927 03:58:24.323973 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:24.323984 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:24.324052 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:24.324210 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:24.324211 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:24.324213 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:58:24.324298 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:24.324300 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:24.324900 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:24.324901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:24.325668 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:24.325670 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:24.326217 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:24.326218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:24.326294 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:24.326297 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:58:24.326360 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 03:58:24.326416 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":2,"thread_start_us":54,"threads_started":1}
I20260927 03:58:24.326824 17513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1afd3ce85d194a9ea6f33c3d4522987e" candidate_uuid: "a07bba424c05479eb82d96d1852e4210" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8beb7b111314835aee34dc4c947a183" is_pre_election: true
I20260927 03:58:24.326915 17513 raft_consensus.cc:2454] T 1afd3ce85d194a9ea6f33c3d4522987e P b8beb7b111314835aee34dc4c947a183 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a07bba424c05479eb82d96d1852e4210 in term 0.
I20260927 03:58:24.327117 17307 leader_election.cc:302] T 1afd3ce85d194a9ea6f33c3d4522987e P a07bba424c05479eb82d96d1852e4210 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a07bba424c05479eb82d96d1852e4210, b8beb7b111314835aee34dc4c947a183; no voters:
I20260927 03:58:24.327142 17653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1afd3ce85d194a9ea6f33c3d4522987e" candidate_uuid: "a07bba424c05479eb82d96d1852e4210" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec4a483880344b1c9602337e8de06098" is_pre_election: true
I20260927 03:58:24.327260 17725 raft_consensus.cc:2781] T 1afd3ce85d194a9ea6f33c3d4522987e P a07bba424c05479eb82d96d1852e4210 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:58:24.327297 17725 raft_consensus.cc:482] T 1afd3ce85d194a9ea6f33c3d4522987e P a07bba424c05479eb82d96d1852e4210 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:58:24.327288 17653 raft_consensus.cc:2454] T 1afd3ce85d194a9ea6f33c3d4522987e P ec4a483880344b1c9602337e8de06098 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a07bba424c05479eb82d96d1852e4210 in term 0.
I20260927 03:58:24.327325 17725 raft_consensus.cc:3037] T 1afd3ce85d194a9ea6f33c3d4522987e P a07bba424c05479eb82d96d1852e4210 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:24.327994 17725 raft_consensus.cc:504] T 1afd3ce85d194a9ea6f33c3d4522987e P a07bba424c05479eb82d96d1852e4210 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07bba424c05479eb82d96d1852e4210" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 40241 } } peers { permanent_uuid: "b8beb7b111314835aee34dc4c947a183" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 39785 } } peers { permanent_uuid: "ec4a483880344b1c9602337e8de06098" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 42711 } }
I20260927 03:58:24.328145 17725 leader_election.cc:288] T 1afd3ce85d194a9ea6f33c3d4522987e P a07bba424c05479eb82d96d1852e4210 [CANDIDATE]: Term 1 election: Requested vote from peers b8beb7b111314835aee34dc4c947a183 (127.15.185.66:39785), ec4a483880344b1c9602337e8de06098 (127.15.185.67:42711)
I20260927 03:58:24.328258 17513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1afd3ce85d194a9ea6f33c3d4522987e" candidate_uuid: "a07bba424c05479eb82d96d1852e4210" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8beb7b111314835aee34dc4c947a183"
I20260927 03:58:24.328346 17513 raft_consensus.cc:3037] T 1afd3ce85d194a9ea6f33c3d4522987e P b8beb7b111314835aee34dc4c947a183 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:24.328315 17653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1afd3ce85d194a9ea6f33c3d4522987e" candidate_uuid: "a07bba424c05479eb82d96d1852e4210" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec4a483880344b1c9602337e8de06098"
I20260927 03:58:24.328408 17653 raft_consensus.cc:3037] T 1afd3ce85d194a9ea6f33c3d4522987e P ec4a483880344b1c9602337e8de06098 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:24.328910 17513 raft_consensus.cc:2454] T 1afd3ce85d194a9ea6f33c3d4522987e P b8beb7b111314835aee34dc4c947a183 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a07bba424c05479eb82d96d1852e4210 in term 1.
I20260927 03:58:24.329061 17307 leader_election.cc:302] T 1afd3ce85d194a9ea6f33c3d4522987e P a07bba424c05479eb82d96d1852e4210 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a07bba424c05479eb82d96d1852e4210, b8beb7b111314835aee34dc4c947a183; no voters:
I20260927 03:58:24.329154 17725 raft_consensus.cc:2781] T 1afd3ce85d194a9ea6f33c3d4522987e P a07bba424c05479eb82d96d1852e4210 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:58:24.329217 17653 raft_consensus.cc:2454] T 1afd3ce85d194a9ea6f33c3d4522987e P ec4a483880344b1c9602337e8de06098 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a07bba424c05479eb82d96d1852e4210 in term 1.
I20260927 03:58:24.329277 17725 raft_consensus.cc:686] T 1afd3ce85d194a9ea6f33c3d4522987e P a07bba424c05479eb82d96d1852e4210 [term 1 LEADER]: Becoming Leader. State: Replica: a07bba424c05479eb82d96d1852e4210, State: Running, Role: LEADER
I20260927 03:58:24.329347 17725 consensus_queue.cc:237] T 1afd3ce85d194a9ea6f33c3d4522987e P a07bba424c05479eb82d96d1852e4210 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07bba424c05479eb82d96d1852e4210" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 40241 } } peers { permanent_uuid: "b8beb7b111314835aee34dc4c947a183" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 39785 } } peers { permanent_uuid: "ec4a483880344b1c9602337e8de06098" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 42711 } }
I20260927 03:58:24.330073 17216 catalog_manager.cc:5495] T 1afd3ce85d194a9ea6f33c3d4522987e P a07bba424c05479eb82d96d1852e4210 reported cstate change: term changed from 0 to 1, leader changed from <none> to a07bba424c05479eb82d96d1852e4210 (127.15.185.65). New cstate: current_term: 1 leader_uuid: "a07bba424c05479eb82d96d1852e4210" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07bba424c05479eb82d96d1852e4210" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 40241 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8beb7b111314835aee34dc4c947a183" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 39785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec4a483880344b1c9602337e8de06098" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 42711 } health_report { overall_health: UNKNOWN } } }
Sep 27 03:58:24 dist-test-slave-glgz krb5kdc[17171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481503, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.126@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 03:58:24.386612 17267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:24.379885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49238 (local address 127.15.185.126:44897)
0927 03:58:24.380008 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:24.380010 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:24.380072 (+ 62us) server_negotiation.cc:407] Connection header received
0927 03:58:24.380178 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:24.380179 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:24.380190 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:24.380267 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:24.380721 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:24.381204 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:24.381893 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:24.382074 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:24.382177 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:24.384246 (+ 2069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:24.384248 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:24.384248 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:24.384255 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:24.385422 (+ 1167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:24.385812 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:24.385813 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:24.385814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:24.385869 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:24.386057 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:24.386058 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:24.386058 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:24.386165 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:24.386236 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:24.386392 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 03:58:24.386472 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260927 03:58:24.386665 17757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:24.379791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:44897 (local address 127.0.0.1:49238)
0927 03:58:24.379981 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:24.379995 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:24.380113 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:24.380271 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:24.380272 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:24.380590 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:24.380634 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:24.380636 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:24.381288 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:24.381289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:24.381800 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:24.381801 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:24.382177 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:24.382178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:24.382226 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:24.382728 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:24.382732 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:24.384141 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:24.385522 (+ 1381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:24.385530 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:24.385532 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:24.385725 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:24.385934 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:24.385935 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:24.385935 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:24.385995 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:24.386238 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:24.386245 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:24.386289 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 03:58:24.386370 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 03:58:24.386526 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":59,"thread_start_us":98,"threads_started":1}
I20260927 03:58:24.388895 17216 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:49238:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 03:58:24.389097 17216 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:58:24.390663 17353 tablet_service.cc:1467] Processing CreateTablet for tablet 0684e7181ee94de2bc0ed2ed77e7ae07 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=98975fad3d55467c814508eecee3bd36]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 03:58:24.390687 17493 tablet_service.cc:1467] Processing CreateTablet for tablet 0684e7181ee94de2bc0ed2ed77e7ae07 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=98975fad3d55467c814508eecee3bd36]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 03:58:24.390698 17633 tablet_service.cc:1467] Processing CreateTablet for tablet 0684e7181ee94de2bc0ed2ed77e7ae07 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=98975fad3d55467c814508eecee3bd36]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 03:58:24.390841 17353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0684e7181ee94de2bc0ed2ed77e7ae07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:24.390861 17493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0684e7181ee94de2bc0ed2ed77e7ae07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:24.390848 17633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0684e7181ee94de2bc0ed2ed77e7ae07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:24.392421 17721 tablet_bootstrap.cc:492] T 0684e7181ee94de2bc0ed2ed77e7ae07 P b8beb7b111314835aee34dc4c947a183: Bootstrap starting.
I20260927 03:58:24.392534 17720 tablet_bootstrap.cc:492] T 0684e7181ee94de2bc0ed2ed77e7ae07 P ec4a483880344b1c9602337e8de06098: Bootstrap starting.
I20260927 03:58:24.392881 17721 tablet_bootstrap.cc:654] T 0684e7181ee94de2bc0ed2ed77e7ae07 P b8beb7b111314835aee34dc4c947a183: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:24.392891 17720 tablet_bootstrap.cc:654] T 0684e7181ee94de2bc0ed2ed77e7ae07 P ec4a483880344b1c9602337e8de06098: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:24.393153 17719 tablet_bootstrap.cc:492] T 0684e7181ee94de2bc0ed2ed77e7ae07 P a07bba424c05479eb82d96d1852e4210: Bootstrap starting.
I20260927 03:58:24.393533 17720 tablet_bootstrap.cc:492] T 0684e7181ee94de2bc0ed2ed77e7ae07 P ec4a483880344b1c9602337e8de06098: No bootstrap required, opened a new log
I20260927 03:58:24.393580 17720 ts_tablet_manager.cc:1397] T 0684e7181ee94de2bc0ed2ed77e7ae07 P ec4a483880344b1c9602337e8de06098: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 03:58:24.393607 17719 tablet_bootstrap.cc:654] T 0684e7181ee94de2bc0ed2ed77e7ae07 P a07bba424c05479eb82d96d1852e4210: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:24.393741 17720 raft_consensus.cc:348] T 0684e7181ee94de2bc0ed2ed77e7ae07 P ec4a483880344b1c9602337e8de06098 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07bba424c05479eb82d96d1852e4210" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 40241 } } peers { permanent_uuid: "b8beb7b111314835aee34dc4c947a183" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 39785 } } peers { permanent_uuid: "ec4a483880344b1c9602337e8de06098" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 42711 } }
I20260927 03:58:24.393819 17720 raft_consensus.cc:374] T 0684e7181ee94de2bc0ed2ed77e7ae07 P ec4a483880344b1c9602337e8de06098 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:24.393846 17720 raft_consensus.cc:729] T 0684e7181ee94de2bc0ed2ed77e7ae07 P ec4a483880344b1c9602337e8de06098 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec4a483880344b1c9602337e8de06098, State: Initialized, Role: FOLLOWER
I20260927 03:58:24.393913 17720 consensus_queue.cc:260] T 0684e7181ee94de2bc0ed2ed77e7ae07 P ec4a483880344b1c9602337e8de06098 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07bba424c05479eb82d96d1852e4210" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 40241 } } peers { permanent_uuid: "b8beb7b111314835aee34dc4c947a183" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 39785 } } peers { permanent_uuid: "ec4a483880344b1c9602337e8de06098" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 42711 } }
I20260927 03:58:24.394021 17720 ts_tablet_manager.cc:1428] T 0684e7181ee94de2bc0ed2ed77e7ae07 P ec4a483880344b1c9602337e8de06098: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260927 03:58:24.394132 17738 tablet_replica.cc:439] T 0684e7181ee94de2bc0ed2ed77e7ae07 P ec4a483880344b1c9602337e8de06098: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07bba424c05479eb82d96d1852e4210" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 40241 } } peers { permanent_uuid: "b8beb7b111314835aee34dc4c947a183" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 39785 } } peers { permanent_uuid: "ec4a483880344b1c9602337e8de06098" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 42711 } } }
I20260927 03:58:24.394204 17738 tablet_replica.cc:442] T 0684e7181ee94de2bc0ed2ed77e7ae07 P ec4a483880344b1c9602337e8de06098: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:58:24.394384 17719 tablet_bootstrap.cc:492] T 0684e7181ee94de2bc0ed2ed77e7ae07 P a07bba424c05479eb82d96d1852e4210: No bootstrap required, opened a new log
I20260927 03:58:24.394441 17719 ts_tablet_manager.cc:1397] T 0684e7181ee94de2bc0ed2ed77e7ae07 P a07bba424c05479eb82d96d1852e4210: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 03:58:24.394608 17719 raft_consensus.cc:348] T 0684e7181ee94de2bc0ed2ed77e7ae07 P a07bba424c05479eb82d96d1852e4210 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07bba424c05479eb82d96d1852e4210" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 40241 } } peers { permanent_uuid: "b8beb7b111314835aee34dc4c947a183" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 39785 } } peers { permanent_uuid: "ec4a483880344b1c9602337e8de06098" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 42711 } }
I20260927 03:58:24.394677 17719 raft_consensus.cc:374] T 0684e7181ee94de2bc0ed2ed77e7ae07 P a07bba424c05479eb82d96d1852e4210 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:24.394692 17719 raft_consensus.cc:729] T 0684e7181ee94de2bc0ed2ed77e7ae07 P a07bba424c05479eb82d96d1852e4210 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a07bba424c05479eb82d96d1852e4210, State: Initialized, Role: FOLLOWER
I20260927 03:58:24.394752 17719 consensus_queue.cc:260] T 0684e7181ee94de2bc0ed2ed77e7ae07 P a07bba424c05479eb82d96d1852e4210 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07bba424c05479eb82d96d1852e4210" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 40241 } } peers { permanent_uuid: "b8beb7b111314835aee34dc4c947a183" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 39785 } } peers { permanent_uuid: "ec4a483880344b1c9602337e8de06098" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 42711 } }
I20260927 03:58:24.394848 17719 ts_tablet_manager.cc:1428] T 0684e7181ee94de2bc0ed2ed77e7ae07 P a07bba424c05479eb82d96d1852e4210: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:24.394929 17725 tablet_replica.cc:439] T 0684e7181ee94de2bc0ed2ed77e7ae07 P a07bba424c05479eb82d96d1852e4210: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07bba424c05479eb82d96d1852e4210" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 40241 } } peers { permanent_uuid: "b8beb7b111314835aee34dc4c947a183" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 39785 } } peers { permanent_uuid: "ec4a483880344b1c9602337e8de06098" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 42711 } } }
I20260927 03:58:24.395035 17725 tablet_replica.cc:442] T 0684e7181ee94de2bc0ed2ed77e7ae07 P a07bba424c05479eb82d96d1852e4210: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:58:24.395295 17721 tablet_bootstrap.cc:492] T 0684e7181ee94de2bc0ed2ed77e7ae07 P b8beb7b111314835aee34dc4c947a183: No bootstrap required, opened a new log
I20260927 03:58:24.395357 17721 ts_tablet_manager.cc:1397] T 0684e7181ee94de2bc0ed2ed77e7ae07 P b8beb7b111314835aee34dc4c947a183: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260927 03:58:24.395560 17721 raft_consensus.cc:348] T 0684e7181ee94de2bc0ed2ed77e7ae07 P b8beb7b111314835aee34dc4c947a183 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07bba424c05479eb82d96d1852e4210" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 40241 } } peers { permanent_uuid: "b8beb7b111314835aee34dc4c947a183" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 39785 } } peers { permanent_uuid: "ec4a483880344b1c9602337e8de06098" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 42711 } }
I20260927 03:58:24.395661 17721 raft_consensus.cc:374] T 0684e7181ee94de2bc0ed2ed77e7ae07 P b8beb7b111314835aee34dc4c947a183 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:24.395697 17721 raft_consensus.cc:729] T 0684e7181ee94de2bc0ed2ed77e7ae07 P b8beb7b111314835aee34dc4c947a183 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8beb7b111314835aee34dc4c947a183, State: Initialized, Role: FOLLOWER
I20260927 03:58:24.395740 17721 consensus_queue.cc:260] T 0684e7181ee94de2bc0ed2ed77e7ae07 P b8beb7b111314835aee34dc4c947a183 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07bba424c05479eb82d96d1852e4210" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 40241 } } peers { permanent_uuid: "b8beb7b111314835aee34dc4c947a183" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 39785 } } peers { permanent_uuid: "ec4a483880344b1c9602337e8de06098" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 42711 } }
I20260927 03:58:24.395859 17721 ts_tablet_manager.cc:1428] T 0684e7181ee94de2bc0ed2ed77e7ae07 P b8beb7b111314835aee34dc4c947a183: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:24.395910 17726 tablet_replica.cc:439] T 0684e7181ee94de2bc0ed2ed77e7ae07 P b8beb7b111314835aee34dc4c947a183: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07bba424c05479eb82d96d1852e4210" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 40241 } } peers { permanent_uuid: "b8beb7b111314835aee34dc4c947a183" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 39785 } } peers { permanent_uuid: "ec4a483880344b1c9602337e8de06098" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 42711 } } }
I20260927 03:58:24.396032 17726 tablet_replica.cc:442] T 0684e7181ee94de2bc0ed2ed77e7ae07 P b8beb7b111314835aee34dc4c947a183: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:58:24.437230 17738 raft_consensus.cc:482] T 0684e7181ee94de2bc0ed2ed77e7ae07 P ec4a483880344b1c9602337e8de06098 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:58:24.437356 17738 raft_consensus.cc:504] T 0684e7181ee94de2bc0ed2ed77e7ae07 P ec4a483880344b1c9602337e8de06098 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07bba424c05479eb82d96d1852e4210" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 40241 } } peers { permanent_uuid: "b8beb7b111314835aee34dc4c947a183" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 39785 } } peers { permanent_uuid: "ec4a483880344b1c9602337e8de06098" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 42711 } }
I20260927 03:58:24.437580 17738 leader_election.cc:288] T 0684e7181ee94de2bc0ed2ed77e7ae07 P ec4a483880344b1c9602337e8de06098 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a07bba424c05479eb82d96d1852e4210 (127.15.185.65:40241), b8beb7b111314835aee34dc4c947a183 (127.15.185.66:39785)
I20260927 03:58:24.437795 17372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0684e7181ee94de2bc0ed2ed77e7ae07" candidate_uuid: "ec4a483880344b1c9602337e8de06098" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a07bba424c05479eb82d96d1852e4210" is_pre_election: true
I20260927 03:58:24.437799 17513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0684e7181ee94de2bc0ed2ed77e7ae07" candidate_uuid: "ec4a483880344b1c9602337e8de06098" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8beb7b111314835aee34dc4c947a183" is_pre_election: true
I20260927 03:58:24.437897 17372 raft_consensus.cc:2454] T 0684e7181ee94de2bc0ed2ed77e7ae07 P a07bba424c05479eb82d96d1852e4210 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec4a483880344b1c9602337e8de06098 in term 0.
I20260927 03:58:24.437918 17513 raft_consensus.cc:2454] T 0684e7181ee94de2bc0ed2ed77e7ae07 P b8beb7b111314835aee34dc4c947a183 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec4a483880344b1c9602337e8de06098 in term 0.
I20260927 03:58:24.438057 17584 leader_election.cc:302] T 0684e7181ee94de2bc0ed2ed77e7ae07 P ec4a483880344b1c9602337e8de06098 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a07bba424c05479eb82d96d1852e4210, ec4a483880344b1c9602337e8de06098; no voters:
I20260927 03:58:24.438164 17738 raft_consensus.cc:2781] T 0684e7181ee94de2bc0ed2ed77e7ae07 P ec4a483880344b1c9602337e8de06098 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:58:24.438218 17738 raft_consensus.cc:482] T 0684e7181ee94de2bc0ed2ed77e7ae07 P ec4a483880344b1c9602337e8de06098 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:58:24.438251 17738 raft_consensus.cc:3037] T 0684e7181ee94de2bc0ed2ed77e7ae07 P ec4a483880344b1c9602337e8de06098 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:24.439023 17738 raft_consensus.cc:504] T 0684e7181ee94de2bc0ed2ed77e7ae07 P ec4a483880344b1c9602337e8de06098 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07bba424c05479eb82d96d1852e4210" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 40241 } } peers { permanent_uuid: "b8beb7b111314835aee34dc4c947a183" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 39785 } } peers { permanent_uuid: "ec4a483880344b1c9602337e8de06098" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 42711 } }
I20260927 03:58:24.439227 17738 leader_election.cc:288] T 0684e7181ee94de2bc0ed2ed77e7ae07 P ec4a483880344b1c9602337e8de06098 [CANDIDATE]: Term 1 election: Requested vote from peers a07bba424c05479eb82d96d1852e4210 (127.15.185.65:40241), b8beb7b111314835aee34dc4c947a183 (127.15.185.66:39785)
I20260927 03:58:24.439389 17513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0684e7181ee94de2bc0ed2ed77e7ae07" candidate_uuid: "ec4a483880344b1c9602337e8de06098" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8beb7b111314835aee34dc4c947a183"
I20260927 03:58:24.439412 17372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0684e7181ee94de2bc0ed2ed77e7ae07" candidate_uuid: "ec4a483880344b1c9602337e8de06098" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a07bba424c05479eb82d96d1852e4210"
I20260927 03:58:24.439467 17513 raft_consensus.cc:3037] T 0684e7181ee94de2bc0ed2ed77e7ae07 P b8beb7b111314835aee34dc4c947a183 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:24.439494 17372 raft_consensus.cc:3037] T 0684e7181ee94de2bc0ed2ed77e7ae07 P a07bba424c05479eb82d96d1852e4210 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:24.440192 17372 raft_consensus.cc:2454] T 0684e7181ee94de2bc0ed2ed77e7ae07 P a07bba424c05479eb82d96d1852e4210 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec4a483880344b1c9602337e8de06098 in term 1.
I20260927 03:58:24.440239 17513 raft_consensus.cc:2454] T 0684e7181ee94de2bc0ed2ed77e7ae07 P b8beb7b111314835aee34dc4c947a183 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec4a483880344b1c9602337e8de06098 in term 1.
I20260927 03:58:24.440387 17587 leader_election.cc:302] T 0684e7181ee94de2bc0ed2ed77e7ae07 P ec4a483880344b1c9602337e8de06098 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b8beb7b111314835aee34dc4c947a183, ec4a483880344b1c9602337e8de06098; no voters:
I20260927 03:58:24.440510 17738 raft_consensus.cc:2781] T 0684e7181ee94de2bc0ed2ed77e7ae07 P ec4a483880344b1c9602337e8de06098 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:58:24.440603 17738 raft_consensus.cc:686] T 0684e7181ee94de2bc0ed2ed77e7ae07 P ec4a483880344b1c9602337e8de06098 [term 1 LEADER]: Becoming Leader. State: Replica: ec4a483880344b1c9602337e8de06098, State: Running, Role: LEADER
I20260927 03:58:24.440662 17738 consensus_queue.cc:237] T 0684e7181ee94de2bc0ed2ed77e7ae07 P ec4a483880344b1c9602337e8de06098 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07bba424c05479eb82d96d1852e4210" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 40241 } } peers { permanent_uuid: "b8beb7b111314835aee34dc4c947a183" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 39785 } } peers { permanent_uuid: "ec4a483880344b1c9602337e8de06098" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 42711 } }
I20260927 03:58:24.440907 17727 tablet_replica.cc:439] T 0684e7181ee94de2bc0ed2ed77e7ae07 P ec4a483880344b1c9602337e8de06098: TxnStatusTablet state changed. Reason: New leader ec4a483880344b1c9602337e8de06098. Latest consensus state: current_term: 1 leader_uuid: "ec4a483880344b1c9602337e8de06098" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07bba424c05479eb82d96d1852e4210" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 40241 } } peers { permanent_uuid: "b8beb7b111314835aee34dc4c947a183" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 39785 } } peers { permanent_uuid: "ec4a483880344b1c9602337e8de06098" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 42711 } } }
I20260927 03:58:24.441051 17727 tablet_replica.cc:442] T 0684e7181ee94de2bc0ed2ed77e7ae07 P ec4a483880344b1c9602337e8de06098: This TxnStatusTablet replica's current role is: LEADER
I20260927 03:58:24.441210 17762 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 03:58:24.441227 17216 catalog_manager.cc:5495] T 0684e7181ee94de2bc0ed2ed77e7ae07 P ec4a483880344b1c9602337e8de06098 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec4a483880344b1c9602337e8de06098 (127.15.185.67). New cstate: current_term: 1 leader_uuid: "ec4a483880344b1c9602337e8de06098" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07bba424c05479eb82d96d1852e4210" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 40241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8beb7b111314835aee34dc4c947a183" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 39785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec4a483880344b1c9602337e8de06098" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 42711 } health_report { overall_health: HEALTHY } } }
I20260927 03:58:24.441354 17762 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 03:58:24.453959 17757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:24.451136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.67:42711 (local address 127.0.0.1:53246)
0927 03:58:24.451259 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:24.451269 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:24.451315 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:24.451457 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:24.451458 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:24.451459 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 03:58:24.451502 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:24.451504 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:24.452425 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:24.452426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:24.453078 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:24.453081 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:24.453544 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:24.453545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:24.453592 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:24.453609 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 03:58:24.453791 (+ 182us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 03:58:24.453798 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 03:58:24.453833 (+ 35us) client_negotiation.cc:240] Negotiation successful
0927 03:58:24.453894 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260927 03:58:24.453956 17717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:24.451192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53246 (local address 127.15.185.67:42711)
0927 03:58:24.451235 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:24.451237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:24.451303 (+ 66us) server_negotiation.cc:407] Connection header received
0927 03:58:24.451380 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:24.451381 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:24.451392 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:24.451444 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:58:24.451571 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:24.452316 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:24.453165 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:24.453461 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:24.453533 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:24.453695 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:58:24.453729 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:58:24.453782 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:24.453844 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 03:58:24.453876 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 03:58:24.455147 17513 raft_consensus.cc:1261] T 0684e7181ee94de2bc0ed2ed77e7ae07 P b8beb7b111314835aee34dc4c947a183 [term 1 FOLLOWER]: Refusing update from remote peer ec4a483880344b1c9602337e8de06098: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:58:24.455175 17372 raft_consensus.cc:1261] T 0684e7181ee94de2bc0ed2ed77e7ae07 P a07bba424c05479eb82d96d1852e4210 [term 1 FOLLOWER]: Refusing update from remote peer ec4a483880344b1c9602337e8de06098: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:58:24.455371 17738 consensus_queue.cc:1035] T 0684e7181ee94de2bc0ed2ed77e7ae07 P ec4a483880344b1c9602337e8de06098 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a07bba424c05479eb82d96d1852e4210" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 40241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:58:24.455473 17727 consensus_queue.cc:1035] T 0684e7181ee94de2bc0ed2ed77e7ae07 P ec4a483880344b1c9602337e8de06098 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8beb7b111314835aee34dc4c947a183" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 39785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:58:24.456044 17725 tablet_replica.cc:439] T 0684e7181ee94de2bc0ed2ed77e7ae07 P a07bba424c05479eb82d96d1852e4210: TxnStatusTablet state changed. Reason: New leader ec4a483880344b1c9602337e8de06098. Latest consensus state: current_term: 1 leader_uuid: "ec4a483880344b1c9602337e8de06098" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07bba424c05479eb82d96d1852e4210" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 40241 } } peers { permanent_uuid: "b8beb7b111314835aee34dc4c947a183" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 39785 } } peers { permanent_uuid: "ec4a483880344b1c9602337e8de06098" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 42711 } } }
I20260927 03:58:24.456137 17725 tablet_replica.cc:442] T 0684e7181ee94de2bc0ed2ed77e7ae07 P a07bba424c05479eb82d96d1852e4210: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:58:24.456079 17726 tablet_replica.cc:439] T 0684e7181ee94de2bc0ed2ed77e7ae07 P b8beb7b111314835aee34dc4c947a183: TxnStatusTablet state changed. Reason: New leader ec4a483880344b1c9602337e8de06098. Latest consensus state: current_term: 1 leader_uuid: "ec4a483880344b1c9602337e8de06098" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07bba424c05479eb82d96d1852e4210" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 40241 } } peers { permanent_uuid: "b8beb7b111314835aee34dc4c947a183" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 39785 } } peers { permanent_uuid: "ec4a483880344b1c9602337e8de06098" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 42711 } } }
I20260927 03:58:24.456166 17726 tablet_replica.cc:442] T 0684e7181ee94de2bc0ed2ed77e7ae07 P b8beb7b111314835aee34dc4c947a183: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:58:24.456812 17738 tablet_replica.cc:439] T 0684e7181ee94de2bc0ed2ed77e7ae07 P ec4a483880344b1c9602337e8de06098: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ec4a483880344b1c9602337e8de06098" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07bba424c05479eb82d96d1852e4210" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 40241 } } peers { permanent_uuid: "b8beb7b111314835aee34dc4c947a183" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 39785 } } peers { permanent_uuid: "ec4a483880344b1c9602337e8de06098" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 42711 } } }
I20260927 03:58:24.456893 17738 tablet_replica.cc:442] T 0684e7181ee94de2bc0ed2ed77e7ae07 P ec4a483880344b1c9602337e8de06098: This TxnStatusTablet replica's current role is: LEADER
I20260927 03:58:24.456951 17732 mvcc.cc:205] Tried to move back new op lower bound from 7333812242246533120 to 7333812242189332480. Current Snapshot: MvccSnapshot[applied={T|T < 7333812242246533120}]
I20260927 03:58:24.457005 17733 mvcc.cc:205] Tried to move back new op lower bound from 7333812242246533120 to 7333812242189332480. Current Snapshot: MvccSnapshot[applied={T|T < 7333812242246533120}]
I20260927 03:58:24.457105 17726 tablet_replica.cc:439] T 0684e7181ee94de2bc0ed2ed77e7ae07 P b8beb7b111314835aee34dc4c947a183: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ec4a483880344b1c9602337e8de06098" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07bba424c05479eb82d96d1852e4210" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 40241 } } peers { permanent_uuid: "b8beb7b111314835aee34dc4c947a183" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 39785 } } peers { permanent_uuid: "ec4a483880344b1c9602337e8de06098" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 42711 } } }
I20260927 03:58:24.457202 17726 tablet_replica.cc:442] T 0684e7181ee94de2bc0ed2ed77e7ae07 P b8beb7b111314835aee34dc4c947a183: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:58:24.457715 17727 tablet_replica.cc:439] T 0684e7181ee94de2bc0ed2ed77e7ae07 P ec4a483880344b1c9602337e8de06098: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ec4a483880344b1c9602337e8de06098" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07bba424c05479eb82d96d1852e4210" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 40241 } } peers { permanent_uuid: "b8beb7b111314835aee34dc4c947a183" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 39785 } } peers { permanent_uuid: "ec4a483880344b1c9602337e8de06098" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 42711 } } }
I20260927 03:58:24.457788 17727 tablet_replica.cc:442] T 0684e7181ee94de2bc0ed2ed77e7ae07 P ec4a483880344b1c9602337e8de06098: This TxnStatusTablet replica's current role is: LEADER
I20260927 03:58:24.458413 17725 tablet_replica.cc:439] T 0684e7181ee94de2bc0ed2ed77e7ae07 P a07bba424c05479eb82d96d1852e4210: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ec4a483880344b1c9602337e8de06098" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a07bba424c05479eb82d96d1852e4210" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 40241 } } peers { permanent_uuid: "b8beb7b111314835aee34dc4c947a183" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 39785 } } peers { permanent_uuid: "ec4a483880344b1c9602337e8de06098" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 42711 } } }
I20260927 03:58:24.458495 17725 tablet_replica.cc:442] T 0684e7181ee94de2bc0ed2ed77e7ae07 P a07bba424c05479eb82d96d1852e4210: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:58:24.461750 17718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:24.458801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39286 (local address 127.15.185.65:40241)
0927 03:58:24.458841 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:24.458843 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:24.458868 (+ 25us) server_negotiation.cc:407] Connection header received
0927 03:58:24.458955 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:24.458956 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:24.458979 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:24.459085 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:58:24.459256 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:24.459815 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:24.460894 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:24.461136 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:24.461220 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:24.461365 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:58:24.461499 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:58:24.461592 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:24.461639 (+ 47us) server_negotiation.cc:299] Negotiation successful
0927 03:58:24.461659 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 03:58:24.467243 17717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:24.464545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:48005 (local address 127.15.185.67:42711)
0927 03:58:24.464591 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:24.464593 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:24.464664 (+ 71us) server_negotiation.cc:407] Connection header received
0927 03:58:24.464740 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:24.464741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:24.464750 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:24.464800 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:58:24.464963 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:24.465483 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:24.466393 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:24.466656 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:24.466736 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:24.466931 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:58:24.466972 (+ 41us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:58:24.467051 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:24.467141 (+ 90us) server_negotiation.cc:299] Negotiation successful
0927 03:58:24.467163 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 03:58:24.467278 17769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:24.464461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.67:42711 (local address 127.15.185.65:48005)
0927 03:58:24.464613 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:24.464623 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:24.464679 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:24.464804 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:24.464805 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:24.464807 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 03:58:24.464873 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:24.464874 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:24.465578 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:24.465580 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:24.466289 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:24.466291 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:24.466733 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:24.466734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:24.466805 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:24.466828 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 03:58:24.467074 (+ 246us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 03:58:24.467076 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 03:58:24.467135 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 03:58:24.467176 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":72,"threads_started":1}
I20260927 03:58:24.470058 17513 raft_consensus.cc:1261] T 1afd3ce85d194a9ea6f33c3d4522987e P b8beb7b111314835aee34dc4c947a183 [term 1 FOLLOWER]: Refusing update from remote peer a07bba424c05479eb82d96d1852e4210: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:58:24.470115 17653 raft_consensus.cc:1261] T 1afd3ce85d194a9ea6f33c3d4522987e P ec4a483880344b1c9602337e8de06098 [term 1 FOLLOWER]: Refusing update from remote peer a07bba424c05479eb82d96d1852e4210: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:58:24.470304 17750 consensus_queue.cc:1035] T 1afd3ce85d194a9ea6f33c3d4522987e P a07bba424c05479eb82d96d1852e4210 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8beb7b111314835aee34dc4c947a183" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 39785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:58:24.470414 17725 consensus_queue.cc:1035] T 1afd3ce85d194a9ea6f33c3d4522987e P a07bba424c05479eb82d96d1852e4210 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec4a483880344b1c9602337e8de06098" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 42711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:58:24.480008 17718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:24.477005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:56547 (local address 127.15.185.65:40241)
0927 03:58:24.477047 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:24.477050 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:24.477199 (+ 149us) server_negotiation.cc:407] Connection header received
0927 03:58:24.477373 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:24.477374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:24.477385 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:24.477446 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:58:24.477599 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:24.478105 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:24.479090 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:24.479407 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:24.479507 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:24.479694 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:58:24.479728 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:58:24.479813 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:24.479881 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 03:58:24.479898 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 03:58:24.480024 17774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:24.476961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.65:40241 (local address 127.15.185.67:56547)
0927 03:58:24.477129 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:24.477143 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:24.477279 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:24.477449 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:24.477450 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:24.477452 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 03:58:24.477513 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:24.477515 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:24.478194 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:24.478195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:24.478971 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:24.478974 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:24.479510 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:24.479511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:24.479572 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:24.479594 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 03:58:24.479816 (+ 222us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 03:58:24.479818 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 03:58:24.479876 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 03:58:24.479922 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":78,"threads_started":1}
I20260927 03:58:24.489631 17214 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:58:24.489780 17214 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:58:24.490617 17214 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b08185cf201e47eebf83a05b40d72b4d: Sending DeleteTablet for 3 replicas of tablet 1afd3ce85d194a9ea6f33c3d4522987e
I20260927 03:58:24.490893 17353 tablet_service.cc:1514] Processing DeleteTablet for tablet 1afd3ce85d194a9ea6f33c3d4522987e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:58:24 UTC) from {username='kudu', principal='kudu/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:39280
I20260927 03:58:24.491019 17633 tablet_service.cc:1514] Processing DeleteTablet for tablet 1afd3ce85d194a9ea6f33c3d4522987e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:58:24 UTC) from {username='kudu', principal='kudu/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:53228
I20260927 03:58:24.491020 17743 tablet_replica.cc:330] T 1afd3ce85d194a9ea6f33c3d4522987e P a07bba424c05479eb82d96d1852e4210: stopping tablet replica
I20260927 03:58:24.491119 17743 raft_consensus.cc:2229] T 1afd3ce85d194a9ea6f33c3d4522987e P a07bba424c05479eb82d96d1852e4210 [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:58:24.491142 17493 tablet_service.cc:1514] Processing DeleteTablet for tablet 1afd3ce85d194a9ea6f33c3d4522987e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:58:24 UTC) from {username='kudu', principal='kudu/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:58534
I20260927 03:58:24.491210 17743 raft_consensus.cc:2258] T 1afd3ce85d194a9ea6f33c3d4522987e P a07bba424c05479eb82d96d1852e4210 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:58:24.491227 17744 tablet_replica.cc:330] T 1afd3ce85d194a9ea6f33c3d4522987e P ec4a483880344b1c9602337e8de06098: stopping tablet replica
I20260927 03:58:24.491285 17744 raft_consensus.cc:2229] T 1afd3ce85d194a9ea6f33c3d4522987e P ec4a483880344b1c9602337e8de06098 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:58:24.491331 17744 raft_consensus.cc:2258] T 1afd3ce85d194a9ea6f33c3d4522987e P ec4a483880344b1c9602337e8de06098 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:58:24.491410 17742 tablet_replica.cc:330] T 1afd3ce85d194a9ea6f33c3d4522987e P b8beb7b111314835aee34dc4c947a183: stopping tablet replica
I20260927 03:58:24.491473 17742 raft_consensus.cc:2229] T 1afd3ce85d194a9ea6f33c3d4522987e P b8beb7b111314835aee34dc4c947a183 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:58:24.491518 17742 raft_consensus.cc:2258] T 1afd3ce85d194a9ea6f33c3d4522987e P b8beb7b111314835aee34dc4c947a183 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:58:24.491549 17743 ts_tablet_manager.cc:1905] T 1afd3ce85d194a9ea6f33c3d4522987e P a07bba424c05479eb82d96d1852e4210: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:58:24.491869 17742 ts_tablet_manager.cc:1905] T 1afd3ce85d194a9ea6f33c3d4522987e P b8beb7b111314835aee34dc4c947a183: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:58:24.491991 17744 ts_tablet_manager.cc:1905] T 1afd3ce85d194a9ea6f33c3d4522987e P ec4a483880344b1c9602337e8de06098: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:58:24.493541 17742 ts_tablet_manager.cc:1918] T 1afd3ce85d194a9ea6f33c3d4522987e P b8beb7b111314835aee34dc4c947a183: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:58:24.493541 17744 ts_tablet_manager.cc:1918] T 1afd3ce85d194a9ea6f33c3d4522987e P ec4a483880344b1c9602337e8de06098: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:58:24.493597 17744 log.cc:1196] T 1afd3ce85d194a9ea6f33c3d4522987e P ec4a483880344b1c9602337e8de06098: Deleting WAL directory at /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/ts-2/wal/wals/1afd3ce85d194a9ea6f33c3d4522987e
I20260927 03:58:24.493597 17742 log.cc:1196] T 1afd3ce85d194a9ea6f33c3d4522987e P b8beb7b111314835aee34dc4c947a183: Deleting WAL directory at /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/ts-1/wal/wals/1afd3ce85d194a9ea6f33c3d4522987e
I20260927 03:58:24.493837 17742 ts_tablet_manager.cc:1939] T 1afd3ce85d194a9ea6f33c3d4522987e P b8beb7b111314835aee34dc4c947a183: Deleting consensus metadata
I20260927 03:58:24.493836 17744 ts_tablet_manager.cc:1939] T 1afd3ce85d194a9ea6f33c3d4522987e P ec4a483880344b1c9602337e8de06098: Deleting consensus metadata
I20260927 03:58:24.494256 17203 catalog_manager.cc:4841] TS b8beb7b111314835aee34dc4c947a183 (127.15.185.66:39785): tablet 1afd3ce85d194a9ea6f33c3d4522987e (table test-table [id=7cb67a97bf774ffa9e37986242ba0b93]) successfully deleted
I20260927 03:58:24.494374 17200 catalog_manager.cc:4841] TS ec4a483880344b1c9602337e8de06098 (127.15.185.67:42711): tablet 1afd3ce85d194a9ea6f33c3d4522987e (table test-table [id=7cb67a97bf774ffa9e37986242ba0b93]) successfully deleted
I20260927 03:58:24.495678 17743 ts_tablet_manager.cc:1918] T 1afd3ce85d194a9ea6f33c3d4522987e P a07bba424c05479eb82d96d1852e4210: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:58:24.495755 17743 log.cc:1196] T 1afd3ce85d194a9ea6f33c3d4522987e P a07bba424c05479eb82d96d1852e4210: Deleting WAL directory at /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481498018407-16101-0/minicluster-data/ts-0/wal/wals/1afd3ce85d194a9ea6f33c3d4522987e
I20260927 03:58:24.496011 17743 ts_tablet_manager.cc:1939] T 1afd3ce85d194a9ea6f33c3d4522987e P a07bba424c05479eb82d96d1852e4210: Deleting consensus metadata
I20260927 03:58:24.496335 17200 catalog_manager.cc:4841] TS a07bba424c05479eb82d96d1852e4210 (127.15.185.65:40241): tablet 1afd3ce85d194a9ea6f33c3d4522987e (table test-table [id=7cb67a97bf774ffa9e37986242ba0b93]) successfully deleted
Sep 27 03:58:24 dist-test-slave-glgz krb5kdc[17171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481503, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.185.65@KRBTEST.COM
I20260927 03:58:24.504287 17718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:24.494909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39290 (local address 127.15.185.65:40241)
0927 03:58:24.494965 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:24.494968 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:24.495167 (+ 199us) server_negotiation.cc:407] Connection header received
0927 03:58:24.495232 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:24.495233 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:24.495245 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:24.495295 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:24.496255 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:24.496828 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:24.497661 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:24.497887 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:24.497972 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:24.501107 (+ 3135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:24.501115 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:24.501115 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:24.501135 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:24.502891 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:24.503338 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:24.503341 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:24.503342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:24.503419 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:24.503650 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:24.503651 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:24.503651 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:24.503799 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:24.503989 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:24.504051 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 03:58:24.504138 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260927 03:58:24.504559 17413 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39290
I20260927 03:58:24.512346 17267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:24.505481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49250 (local address 127.15.185.126:44897)
0927 03:58:24.505531 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:24.505534 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:24.505777 (+ 243us) server_negotiation.cc:407] Connection header received
0927 03:58:24.505862 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:24.505863 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:24.505875 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:24.505925 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:24.506473 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:24.507023 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:24.507841 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:24.508080 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:24.508176 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:24.509432 (+ 1256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:24.509434 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:24.509434 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:24.509442 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:24.511021 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:24.511433 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:24.511434 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:24.511435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:24.511508 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:24.511722 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:24.511724 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:24.511724 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:24.511851 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:24.511968 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:24.512121 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 03:58:24.512202 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260927 03:58:24.512598 17214 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49250
I20260927 03:58:24.513315 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 17288
I20260927 03:58:24.519863 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 17428
I20260927 03:58:24.525185 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 17568
I20260927 03:58:24.531227 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 17184
2026-09-27T03:58:24Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2472 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:58:24 dist-test-slave-glgz krb5kdc[17783](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:58:24 dist-test-slave-glgz krb5kdc[17783](info): set up 2 sockets
Sep 27 03:58:24 dist-test-slave-glgz krb5kdc[17783](info): commencing operation
krb5kdc: starting...
W20260927 03:58:25.571228 16101 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Sep 27 03:58:25 dist-test-slave-glgz krb5kdc[17783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:58:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:58:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.185.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab.
Entry for principal kudu/127.15.185.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab.
I20260927 03:58:25.690817 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:41063
--webserver_interface=127.15.185.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:45085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.185.126:41063
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:58:25.769498 17796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:25.769763 17796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:25.769799 17796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:25.770951 17796 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:58:25.771023 17796 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 03:58:25.771044 17796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:25.771063 17796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:25.771127 17796 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:58:25.771152 17796 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:58:25.772622 17796 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:45085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.185.126:41063
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:41063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17796
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:25.772887 17796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:25.773114 17796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:25.777890 17801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:25.777841 17802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:25.777869 17796 server_base.cc:1034] running on GCE node
W20260927 03:58:25.777844 17804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:25.778365 17796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:25.778604 17796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:25.779749 17796 hybrid_clock.cc:648] HybridClock initialized: now 1790481505779726 us; error 31 us; skew 500 ppm
Sep 27 03:58:25 dist-test-slave-glgz krb5kdc[17783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481505, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:25.782132 17796 init.cc:377] Logged in from keytab as kudu/127.15.185.126@KRBTEST.COM (short username kudu)
I20260927 03:58:25.782567 17796 webserver.cc:460] Webserver started at http://127.15.185.126:44767/ using document root <none> and password file <none>
I20260927 03:58:25.782795 17796 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:25.782860 17796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:25.783004 17796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:25.784042 17796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/master-0/data/instance:
uuid: "67a6d9bd1a4e47818b720084e11ccd8a"
format_stamp: "Formatted at 2026-09-27 03:58:25 on dist-test-slave-glgz"
server_key: "b2f80dbe4cbfa08832361494cd880532"
server_key_iv: "af5ff1e5a861ba811c33b7a91e16aca4"
server_key_version: "encryptionkey@0"
I20260927 03:58:25.784390 17796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/master-0/wal/instance:
uuid: "67a6d9bd1a4e47818b720084e11ccd8a"
format_stamp: "Formatted at 2026-09-27 03:58:25 on dist-test-slave-glgz"
server_key: "b2f80dbe4cbfa08832361494cd880532"
server_key_iv: "af5ff1e5a861ba811c33b7a91e16aca4"
server_key_version: "encryptionkey@0"
I20260927 03:58:25.785755 17796 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:58:25.786427 17811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:25.786620 17796 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:58:25.786726 17796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/master-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/master-0/wal
uuid: "67a6d9bd1a4e47818b720084e11ccd8a"
format_stamp: "Formatted at 2026-09-27 03:58:25 on dist-test-slave-glgz"
server_key: "b2f80dbe4cbfa08832361494cd880532"
server_key_iv: "af5ff1e5a861ba811c33b7a91e16aca4"
server_key_version: "encryptionkey@0"
I20260927 03:58:25.786808 17796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:25.796573 17796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:25.798436 17796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:25.798594 17796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:25.803112 17796 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.126:41063
I20260927 03:58:25.803164 17873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.126:41063 every 8 connection(s)
I20260927 03:58:25.803512 17796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
I20260927 03:58:25.804113 17874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:25.805275 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 17796
I20260927 03:58:25.805357 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/master-0/wal/instance
I20260927 03:58:25.805572 16101 external_mini_cluster.cc:1428] Setting key 98d2279466958aa2181c3ebee7a22f18
I20260927 03:58:25.806742 17874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67a6d9bd1a4e47818b720084e11ccd8a: Bootstrap starting.
I20260927 03:58:25.807529 17874 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67a6d9bd1a4e47818b720084e11ccd8a: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:25.807972 17874 log.cc:824] T 00000000000000000000000000000000 P 67a6d9bd1a4e47818b720084e11ccd8a: Log is configured to *not* fsync() on all Append() calls
I20260927 03:58:25.808748 17874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67a6d9bd1a4e47818b720084e11ccd8a: No bootstrap required, opened a new log
I20260927 03:58:25.811033 17874 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67a6d9bd1a4e47818b720084e11ccd8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67a6d9bd1a4e47818b720084e11ccd8a" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 41063 } }
I20260927 03:58:25.811265 17874 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67a6d9bd1a4e47818b720084e11ccd8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:25.811303 17874 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67a6d9bd1a4e47818b720084e11ccd8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67a6d9bd1a4e47818b720084e11ccd8a, State: Initialized, Role: FOLLOWER
I20260927 03:58:25.811440 17874 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67a6d9bd1a4e47818b720084e11ccd8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67a6d9bd1a4e47818b720084e11ccd8a" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 41063 } }
I20260927 03:58:25.811527 17874 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67a6d9bd1a4e47818b720084e11ccd8a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:58:25.811563 17874 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67a6d9bd1a4e47818b720084e11ccd8a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:58:25.811630 17874 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67a6d9bd1a4e47818b720084e11ccd8a [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:25.812367 17874 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67a6d9bd1a4e47818b720084e11ccd8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67a6d9bd1a4e47818b720084e11ccd8a" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 41063 } }
I20260927 03:58:25.812502 17874 leader_election.cc:302] T 00000000000000000000000000000000 P 67a6d9bd1a4e47818b720084e11ccd8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67a6d9bd1a4e47818b720084e11ccd8a; no voters:
I20260927 03:58:25.812681 17874 leader_election.cc:288] T 00000000000000000000000000000000 P 67a6d9bd1a4e47818b720084e11ccd8a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:58:25.812704 17879 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67a6d9bd1a4e47818b720084e11ccd8a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:58:25.812814 17879 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67a6d9bd1a4e47818b720084e11ccd8a [term 1 LEADER]: Becoming Leader. State: Replica: 67a6d9bd1a4e47818b720084e11ccd8a, State: Running, Role: LEADER
I20260927 03:58:25.812896 17879 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67a6d9bd1a4e47818b720084e11ccd8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67a6d9bd1a4e47818b720084e11ccd8a" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 41063 } }
I20260927 03:58:25.812932 17874 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67a6d9bd1a4e47818b720084e11ccd8a [sys.catalog]: configured and running, proceeding with master startup.
Sep 27 03:58:25 dist-test-slave-glgz krb5kdc[17783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 03:58:25.813265 17879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67a6d9bd1a4e47818b720084e11ccd8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67a6d9bd1a4e47818b720084e11ccd8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67a6d9bd1a4e47818b720084e11ccd8a" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 41063 } } }
I20260927 03:58:25.813336 17879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67a6d9bd1a4e47818b720084e11ccd8a [sys.catalog]: This master's current role is: LEADER
I20260927 03:58:25.813450 17879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67a6d9bd1a4e47818b720084e11ccd8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67a6d9bd1a4e47818b720084e11ccd8a. Latest consensus state: current_term: 1 leader_uuid: "67a6d9bd1a4e47818b720084e11ccd8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67a6d9bd1a4e47818b720084e11ccd8a" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 41063 } } }
I20260927 03:58:25.813508 17879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67a6d9bd1a4e47818b720084e11ccd8a [sys.catalog]: This master's current role is: LEADER
W20260927 03:58:25.815186 17895 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 67a6d9bd1a4e47818b720084e11ccd8a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:58:25.815284 17895 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:58:25.815405 17884 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:58:25.815639 17884 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:58:25.817582 17884 catalog_manager.cc:1347] Generated new cluster ID: 2ca64df4d95a45a3b1ca0c007c0aca52
I20260927 03:58:25.817667 17884 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:58:25.817742 17877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:25.806163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36514 (local address 127.15.185.126:41063)
0927 03:58:25.806340 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:25.806348 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:25.806372 (+ 24us) server_negotiation.cc:407] Connection header received
0927 03:58:25.806747 (+ 375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:25.806754 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:25.806797 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:25.806951 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:25.807809 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:25.809019 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:25.810193 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:25.810628 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:25.810718 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:25.813790 (+ 3072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:25.813794 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:25.813794 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:25.813804 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:25.815939 (+ 2135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:25.816363 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:25.816366 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:25.816367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:25.816480 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:25.816696 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:25.816697 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:25.816697 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:25.817001 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:25.817167 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:25.817311 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 03:58:25.817464 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":85,"threads_started":1}
I20260927 03:58:25.822613 17884 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:58:25.823230 17884 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:58:25.827102 17884 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67a6d9bd1a4e47818b720084e11ccd8a: Generated new TSK 0
I20260927 03:58:25.827301 17884 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.185.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab.
Entry for principal kudu/127.15.185.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab.
I20260927 03:58:25.858557 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.65:0
--local_ip_for_outbound_sockets=127.15.185.65
--webserver_interface=127.15.185.65
--webserver_port=0
--tserver_master_addrs=127.15.185.126:41063
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab
--principal=kudu/127.15.185.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:45085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:58:25.947772 17900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:25.948046 17900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:25.948101 17900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:25.949318 17900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:25.949393 17900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:25.949426 17900 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 03:58:25.949456 17900 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 03:58:25.949489 17900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.65
I20260927 03:58:25.950982 17900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:45085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab
--principal=kudu/127.15.185.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.185.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.185.126:41063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.65
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:25.951300 17900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:25.951541 17900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:25.957595 17905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:25.957602 17906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:25.957604 17908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:25.957834 17900 server_base.cc:1034] running on GCE node
I20260927 03:58:25.958091 17900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:25.958359 17900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:25.959535 17900 hybrid_clock.cc:648] HybridClock initialized: now 1790481505959516 us; error 40 us; skew 500 ppm
Sep 27 03:58:25 dist-test-slave-glgz krb5kdc[17783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481505, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:25.962041 17900 init.cc:377] Logged in from keytab as kudu/127.15.185.65@KRBTEST.COM (short username kudu)
I20260927 03:58:25.962559 17900 webserver.cc:460] Webserver started at http://127.15.185.65:38145/ using document root <none> and password file <none>
I20260927 03:58:25.962803 17900 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:25.962848 17900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:25.962984 17900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:25.964126 17900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/ts-0/data/instance:
uuid: "47a4ce643e7f40d4b3ff4892217c5352"
format_stamp: "Formatted at 2026-09-27 03:58:25 on dist-test-slave-glgz"
server_key: "81b11056d66f8f31a32f968da2c9480c"
server_key_iv: "227e2a7462c31daf6c5689d2b1ee7ddd"
server_key_version: "encryptionkey@0"
I20260927 03:58:25.964489 17900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/ts-0/wal/instance:
uuid: "47a4ce643e7f40d4b3ff4892217c5352"
format_stamp: "Formatted at 2026-09-27 03:58:25 on dist-test-slave-glgz"
server_key: "81b11056d66f8f31a32f968da2c9480c"
server_key_iv: "227e2a7462c31daf6c5689d2b1ee7ddd"
server_key_version: "encryptionkey@0"
I20260927 03:58:25.965930 17900 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:58:25.966760 17915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:25.966958 17900 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:58:25.967090 17900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/ts-0/wal
uuid: "47a4ce643e7f40d4b3ff4892217c5352"
format_stamp: "Formatted at 2026-09-27 03:58:25 on dist-test-slave-glgz"
server_key: "81b11056d66f8f31a32f968da2c9480c"
server_key_iv: "227e2a7462c31daf6c5689d2b1ee7ddd"
server_key_version: "encryptionkey@0"
I20260927 03:58:25.967167 17900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:25.978093 17900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:25.980482 17900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:25.980649 17900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:25.981225 17900 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:58:25.981264 17900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:25.981287 17900 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:58:25.981299 17900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 03:58:25 dist-test-slave-glgz krb5kdc[17783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481505, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.65@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 03:58:25.988667 17900 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.65:45357
I20260927 03:58:25.989017 17900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
I20260927 03:58:25.989151 18030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.65:45357 every 8 connection(s)
I20260927 03:58:25.990983 17877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:25.981619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:44279 (local address 127.15.185.126:41063)
0927 03:58:25.981693 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:25.981695 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:25.981910 (+ 215us) server_negotiation.cc:407] Connection header received
0927 03:58:25.982451 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:25.982452 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:25.982465 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:25.982517 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:25.983307 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:25.983865 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:25.984901 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:25.985131 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:25.985224 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:25.988232 (+ 3008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:25.988233 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:25.988233 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:25.988240 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:25.989933 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:25.990285 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:25.990285 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:25.990286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:25.990338 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:25.990516 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:25.990516 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:25.990517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:25.990624 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:25.990664 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:25.990855 (+ 191us) server_negotiation.cc:299] Negotiation successful
0927 03:58:25.990919 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 03:58:25.991247 17924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:25.981623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:41063 (local address 127.15.185.65:44279)
0927 03:58:25.981792 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:25.981826 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:25.982335 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:25.982591 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:25.982595 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:25.982936 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:25.983207 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:25.983212 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:25.983987 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:25.983988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:25.984805 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:25.984807 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:25.985231 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:25.985232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:25.985370 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:25.985875 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:25.985885 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:25.988140 (+ 2255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:25.990002 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:25.990006 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:25.990007 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:25.990216 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:25.990381 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:25.990381 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:25.990382 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:25.990480 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:25.990662 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:25.990665 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:25.990769 (+ 104us) client_negotiation.cc:769] Sending connection context
0927 03:58:25.990842 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 03:58:25.990955 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":82,"threads_started":1}
I20260927 03:58:25.991734 18031 heartbeater.cc:347] Connected to a master server at 127.15.185.126:41063
I20260927 03:58:25.991811 18031 heartbeater.cc:464] Registering TS with master...
I20260927 03:58:25.991986 18031 heartbeater.cc:511] Master 127.15.185.126:41063 requested a full tablet report, sending...
I20260927 03:58:25.992486 17828 ts_manager.cc:194] Registered new tserver with Master: 47a4ce643e7f40d4b3ff4892217c5352 (127.15.185.65:45357)
I20260927 03:58:25.993489 17828 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.185.65@KRBTEST.COM'} at 127.15.185.65:44279
I20260927 03:58:25.993945 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 17900
I20260927 03:58:25.994046 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/ts-0/wal/instance
I20260927 03:58:25.994246 16101 external_mini_cluster.cc:1428] Setting key ab9b3a7cfc45a51b8905bca788e36226
I20260927 03:58:25.998759 17877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:25.992951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:60011 (local address 127.15.185.126:41063)
0927 03:58:25.992987 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:25.992988 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:25.992997 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:58:25.993057 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:25.993058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:25.993065 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:25.993100 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:25.993413 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:25.993979 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:25.994950 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:25.995239 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:25.995323 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:25.996362 (+ 1039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:25.996363 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:25.996364 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:25.996391 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:25.997686 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:25.998081 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:25.998082 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:25.998082 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:25.998129 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:25.998336 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:25.998337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:25.998337 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:25.998426 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:25.998488 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:25.998642 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 03:58:25.998700 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 03:58:25.998852 18037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:25.992775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:41063 (local address 127.15.185.65:60011)
0927 03:58:25.992927 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:25.992939 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:25.992997 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:25.993112 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:25.993112 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:25.993292 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:25.993350 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:25.993351 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:25.994099 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:25.994101 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:25.994874 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:25.994877 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:25.995323 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:25.995324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:25.995400 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:25.995843 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:25.995850 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:25.996261 (+ 411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:25.997769 (+ 1508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:25.997770 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:25.997771 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:25.998005 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:25.998196 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:25.998197 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:25.998197 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:25.998270 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:25.998487 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:25.998488 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:25.998542 (+ 54us) client_negotiation.cc:769] Sending connection context
0927 03:58:25.998628 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 03:58:25.998735 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":21,"thread_start_us":99,"threads_started":1}
WARNING: no policy specified for kudu/127.15.185.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab.
Entry for principal kudu/127.15.185.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab.
I20260927 03:58:26.023161 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.66:0
--local_ip_for_outbound_sockets=127.15.185.66
--webserver_interface=127.15.185.66
--webserver_port=0
--tserver_master_addrs=127.15.185.126:41063
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab
--principal=kudu/127.15.185.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:45085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:58:26.105443 18040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:26.105652 18040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:26.105677 18040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:26.106819 18040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:26.106863 18040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:26.106884 18040 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 03:58:26.106895 18040 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 03:58:26.106920 18040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.66
I20260927 03:58:26.108316 18040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:45085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab
--principal=kudu/127.15.185.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.185.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.185.126:41063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.66
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:26.108502 18040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:26.108680 18040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:26.113646 18045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:26.113684 18046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:26.113670 18048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:26.113956 18040 server_base.cc:1034] running on GCE node
I20260927 03:58:26.114189 18040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:26.114432 18040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:26.115602 18040 hybrid_clock.cc:648] HybridClock initialized: now 1790481506115598 us; error 37 us; skew 500 ppm
Sep 27 03:58:26 dist-test-slave-glgz krb5kdc[17783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481506, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:26.118017 18040 init.cc:377] Logged in from keytab as kudu/127.15.185.66@KRBTEST.COM (short username kudu)
I20260927 03:58:26.118462 18040 webserver.cc:460] Webserver started at http://127.15.185.66:34701/ using document root <none> and password file <none>
I20260927 03:58:26.118698 18040 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:26.118765 18040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:26.118916 18040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:26.120079 18040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/ts-1/data/instance:
uuid: "be26fcebf49a43259cadc0f2b9c0325a"
format_stamp: "Formatted at 2026-09-27 03:58:26 on dist-test-slave-glgz"
server_key: "b24ac9d96cee7b744b2b0e2107a5ea60"
server_key_iv: "9fb65147e06e12409934c7c03cab8b35"
server_key_version: "encryptionkey@0"
I20260927 03:58:26.120474 18040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/ts-1/wal/instance:
uuid: "be26fcebf49a43259cadc0f2b9c0325a"
format_stamp: "Formatted at 2026-09-27 03:58:26 on dist-test-slave-glgz"
server_key: "b24ac9d96cee7b744b2b0e2107a5ea60"
server_key_iv: "9fb65147e06e12409934c7c03cab8b35"
server_key_version: "encryptionkey@0"
I20260927 03:58:26.121878 18040 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:58:26.122615 18055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:26.122799 18040 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:58:26.122915 18040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/ts-1/wal
uuid: "be26fcebf49a43259cadc0f2b9c0325a"
format_stamp: "Formatted at 2026-09-27 03:58:26 on dist-test-slave-glgz"
server_key: "b24ac9d96cee7b744b2b0e2107a5ea60"
server_key_iv: "9fb65147e06e12409934c7c03cab8b35"
server_key_version: "encryptionkey@0"
I20260927 03:58:26.123009 18040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:26.142733 18040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:26.144886 18040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:26.145179 18040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:26.145807 18040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:58:26.145889 18040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:26.145938 18040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:58:26.145968 18040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 03:58:26 dist-test-slave-glgz krb5kdc[17783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481506, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.66@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 03:58:26.153846 18040 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.66:38903
I20260927 03:58:26.153935 18170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.66:38903 every 8 connection(s)
I20260927 03:58:26.154286 18040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
I20260927 03:58:26.156816 17877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:26.146317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:47571 (local address 127.15.185.126:41063)
0927 03:58:26.146399 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:26.146402 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:26.146580 (+ 178us) server_negotiation.cc:407] Connection header received
0927 03:58:26.147092 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:26.147094 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:26.147106 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:26.147201 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:26.147787 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:26.148795 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:26.150401 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:26.150792 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:26.150898 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:26.153759 (+ 2861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:26.153760 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:26.153760 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:26.153768 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:26.155541 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:26.156014 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:26.156015 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:26.156015 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:26.156071 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:26.156290 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:26.156290 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:26.156291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:26.156396 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:26.156448 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:26.156681 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 03:58:26.156754 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 03:58:26.156952 18064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:26.146292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:41063 (local address 127.15.185.66:47571)
0927 03:58:26.146445 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:26.146486 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:26.146965 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:26.147256 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:26.147260 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:26.147490 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:26.147682 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:26.147686 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:26.148901 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:26.148902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:26.150285 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:26.150288 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:26.150897 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:26.150898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:26.151047 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:26.151647 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:26.151659 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:26.153643 (+ 1984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:26.155625 (+ 1982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:26.155630 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:26.155632 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:26.155841 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:26.156137 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:26.156138 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:26.156138 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:26.156231 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:26.156451 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:26.156455 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:26.156565 (+ 110us) client_negotiation.cc:769] Sending connection context
0927 03:58:26.156651 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 03:58:26.156768 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":89,"threads_started":1}
I20260927 03:58:26.157456 18171 heartbeater.cc:347] Connected to a master server at 127.15.185.126:41063
I20260927 03:58:26.157541 18171 heartbeater.cc:464] Registering TS with master...
I20260927 03:58:26.157740 18171 heartbeater.cc:511] Master 127.15.185.126:41063 requested a full tablet report, sending...
I20260927 03:58:26.158141 17828 ts_manager.cc:194] Registered new tserver with Master: be26fcebf49a43259cadc0f2b9c0325a (127.15.185.66:38903)
I20260927 03:58:26.158548 17828 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.185.66@KRBTEST.COM'} at 127.15.185.66:47571
I20260927 03:58:26.159226 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 18040
I20260927 03:58:26.159332 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/ts-1/wal/instance
I20260927 03:58:26.159552 16101 external_mini_cluster.cc:1428] Setting key 9860e3f346c4515e6101240b2d8fc04a
I20260927 03:58:26.165594 17877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:26.158832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:50109 (local address 127.15.185.126:41063)
0927 03:58:26.158889 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:26.158891 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:26.159474 (+ 583us) server_negotiation.cc:407] Connection header received
0927 03:58:26.159537 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:26.159537 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:26.159546 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:26.159589 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:26.160066 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:26.160596 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:26.161654 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:26.161957 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:26.162032 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:26.163041 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:26.163081 (+ 40us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:26.163081 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:26.163089 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:26.164455 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:26.164854 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:26.164855 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:26.164855 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:26.164904 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:26.165115 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:26.165115 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:26.165116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:26.165240 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:26.165292 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:26.165464 (+ 172us) server_negotiation.cc:299] Negotiation successful
0927 03:58:26.165518 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 03:58:26.165723 18177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:26.158762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:41063 (local address 127.15.185.66:50109)
0927 03:58:26.159389 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:26.159402 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:26.159479 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:26.159605 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:26.159606 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:26.159876 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:26.159956 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:26.159958 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:26.160696 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:26.160697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:26.161568 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:26.161571 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:26.162034 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:26.162035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:26.162093 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:26.162536 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:26.162541 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:26.162961 (+ 420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:26.164537 (+ 1576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:26.164538 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:26.164538 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:26.164779 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:26.164970 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:26.164971 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:26.164971 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:26.165043 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:26.165307 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:26.165308 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:26.165361 (+ 53us) client_negotiation.cc:769] Sending connection context
0927 03:58:26.165459 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 03:58:26.165576 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":575,"threads_started":1}
WARNING: no policy specified for kudu/127.15.185.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab.
Entry for principal kudu/127.15.185.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab.
I20260927 03:58:26.185952 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.67:0
--local_ip_for_outbound_sockets=127.15.185.67
--webserver_interface=127.15.185.67
--webserver_port=0
--tserver_master_addrs=127.15.185.126:41063
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab
--principal=kudu/127.15.185.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:45085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:58:26.269529 18180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:26.269738 18180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:26.269764 18180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:26.271015 18180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:26.271108 18180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:26.271132 18180 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 03:58:26.271144 18180 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 03:58:26.271162 18180 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.67
I20260927 03:58:26.272593 18180 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:45085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab
--principal=kudu/127.15.185.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.185.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.185.126:41063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18180
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.67
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:26.272794 18180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:26.272969 18180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:26.278169 18188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:26.278156 18185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:26.278177 18186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:26.278437 18180 server_base.cc:1034] running on GCE node
I20260927 03:58:26.278641 18180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:26.278879 18180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:26.280042 18180 hybrid_clock.cc:648] HybridClock initialized: now 1790481506280018 us; error 56 us; skew 500 ppm
Sep 27 03:58:26 dist-test-slave-glgz krb5kdc[17783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481506, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:26.282613 18180 init.cc:377] Logged in from keytab as kudu/127.15.185.67@KRBTEST.COM (short username kudu)
I20260927 03:58:26.283026 18180 webserver.cc:460] Webserver started at http://127.15.185.67:35319/ using document root <none> and password file <none>
I20260927 03:58:26.283269 18180 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:26.283314 18180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:26.283409 18180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:26.284405 18180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/ts-2/data/instance:
uuid: "1c62cb4ea2bf45a1830e7048550cf4db"
format_stamp: "Formatted at 2026-09-27 03:58:26 on dist-test-slave-glgz"
server_key: "3b2da4f7167d8c804d80e943d46a9589"
server_key_iv: "b2d26c7334b0b9328a292dc2854738a4"
server_key_version: "encryptionkey@0"
I20260927 03:58:26.284706 18180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/ts-2/wal/instance:
uuid: "1c62cb4ea2bf45a1830e7048550cf4db"
format_stamp: "Formatted at 2026-09-27 03:58:26 on dist-test-slave-glgz"
server_key: "3b2da4f7167d8c804d80e943d46a9589"
server_key_iv: "b2d26c7334b0b9328a292dc2854738a4"
server_key_version: "encryptionkey@0"
I20260927 03:58:26.285960 18180 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:58:26.286679 18195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:26.286890 18180 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:58:26.286988 18180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/ts-2/wal
uuid: "1c62cb4ea2bf45a1830e7048550cf4db"
format_stamp: "Formatted at 2026-09-27 03:58:26 on dist-test-slave-glgz"
server_key: "3b2da4f7167d8c804d80e943d46a9589"
server_key_iv: "b2d26c7334b0b9328a292dc2854738a4"
server_key_version: "encryptionkey@0"
I20260927 03:58:26.287082 18180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:26.304428 18180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:26.306370 18180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:26.306488 18180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:26.306949 18180 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:58:26.306980 18180 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:26.307003 18180 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:58:26.307085 18180 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:26.314978 18180 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.67:36419
I20260927 03:58:26.315178 18310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.67:36419 every 8 connection(s)
I20260927 03:58:26.315517 18180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
Sep 27 03:58:26 dist-test-slave-glgz krb5kdc[17783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481506, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.67@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 03:58:26.319403 17877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:26.307583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:59451 (local address 127.15.185.126:41063)
0927 03:58:26.307625 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:26.307627 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:26.307943 (+ 316us) server_negotiation.cc:407] Connection header received
0927 03:58:26.308526 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:26.308527 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:26.308557 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:26.308621 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:26.309547 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:26.310401 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:26.312182 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:26.312521 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:26.312596 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:26.316175 (+ 3579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:26.316178 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:26.316179 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:26.316187 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:26.317761 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:26.318255 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:26.318256 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:26.318257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:26.318330 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:26.318645 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:26.318647 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:26.318647 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:26.318785 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:26.318873 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:26.319221 (+ 348us) server_negotiation.cc:299] Negotiation successful
0927 03:58:26.319309 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 03:58:26.319634 18204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:26.307591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:41063 (local address 127.15.185.67:59451)
0927 03:58:26.307764 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:26.307801 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:26.308404 (+ 603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:26.308755 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:26.308761 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:26.309148 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:26.309429 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:26.309437 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:26.310506 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:26.310507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:26.312090 (+ 1583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:26.312093 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:26.312613 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:26.312614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:26.313662 (+ 1048us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:26.314245 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:26.314255 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:26.316087 (+ 1832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:26.317893 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:26.317900 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:26.317903 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:26.318162 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:26.318417 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:26.318417 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:26.318418 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:26.318556 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:26.318857 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:26.318868 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:26.319028 (+ 160us) client_negotiation.cc:769] Sending connection context
0927 03:58:26.319199 (+ 171us) client_negotiation.cc:240] Negotiation successful
0927 03:58:26.319373 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":94,"threads_started":1}
I20260927 03:58:26.320127 18311 heartbeater.cc:347] Connected to a master server at 127.15.185.126:41063
I20260927 03:58:26.320230 18311 heartbeater.cc:464] Registering TS with master...
I20260927 03:58:26.320464 18311 heartbeater.cc:511] Master 127.15.185.126:41063 requested a full tablet report, sending...
I20260927 03:58:26.320868 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 18180
I20260927 03:58:26.320955 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/ts-2/wal/instance
I20260927 03:58:26.321127 16101 external_mini_cluster.cc:1428] Setting key 11078edd3c57a6aa67aac369fe40bfa3
I20260927 03:58:26.321311 17827 ts_manager.cc:194] Registered new tserver with Master: 1c62cb4ea2bf45a1830e7048550cf4db (127.15.185.67:36419)
I20260927 03:58:26.321841 17827 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.185.67@KRBTEST.COM'} at 127.15.185.67:59451
I20260927 03:58:26.322031 16101 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:58:26.330132 17877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:26.321404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:39457 (local address 127.15.185.126:41063)
0927 03:58:26.321444 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:26.321446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:26.321699 (+ 253us) server_negotiation.cc:407] Connection header received
0927 03:58:26.321928 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:26.321929 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:26.321938 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:26.321990 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:26.323511 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:26.324050 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:26.325184 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:26.325441 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:26.325500 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:26.326788 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:26.326789 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:26.326790 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:26.326799 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:26.328792 (+ 1993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:26.329257 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:26.329258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:26.329259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:26.329315 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:26.329626 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:26.329627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:26.329627 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:26.329749 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:26.329815 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:26.329984 (+ 169us) server_negotiation.cc:299] Negotiation successful
0927 03:58:26.330051 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 03:58:26.330248 18317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:26.321338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:41063 (local address 127.15.185.67:39457)
0927 03:58:26.321647 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:26.321659 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:26.321713 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:26.321995 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:26.321995 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:26.323343 (+ 1348us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:26.323418 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:26.323421 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:26.324159 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:26.324160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:26.325047 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:26.325050 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:26.325526 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:26.325527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:26.325605 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:26.326095 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:26.326100 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:26.326641 (+ 541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:26.328892 (+ 2251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:26.328893 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:26.328894 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:26.329173 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:26.329381 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:26.329382 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:26.329382 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:26.329467 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:26.329825 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:26.329826 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:26.329881 (+ 55us) client_negotiation.cc:769] Sending connection context
0927 03:58:26.329957 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 03:58:26.330086 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":77,"threads_started":1}
Sep 27 03:58:26 dist-test-slave-glgz krb5kdc[17783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481506, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 03:58:26 dist-test-slave-glgz krb5kdc[17783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481506, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 03:58:26.347139 17877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:26.339942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36520 (local address 127.15.185.126:41063)
0927 03:58:26.339989 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:26.339991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:26.339999 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:58:26.340065 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:26.340066 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:26.340074 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:26.340124 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:26.340757 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:26.341265 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:26.342036 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:26.342235 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:26.342304 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:26.344632 (+ 2328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:26.344633 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:26.344634 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:26.344640 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:26.346104 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:26.346424 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:26.346425 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:26.346425 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:26.346481 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:26.346644 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:26.346645 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:26.346645 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:26.346744 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:26.346818 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:26.346897 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 03:58:26.347003 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 03:58:26.348637 17827 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36520:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:58:26.350091 17827 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:58:26.356927 18328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:26.352427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.66:38903 (local address 127.0.0.1:56438)
0927 03:58:26.352535 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:26.352573 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:26.352660 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:26.353050 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:26.353054 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:26.353057 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:58:26.353128 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:26.353130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:26.354602 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:26.354603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:26.355900 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:26.355902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:26.356642 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:26.356643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:26.356691 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:26.356705 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 03:58:26.356752 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 03:58:26.356805 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"mutex_wait_us":71,"thread_start_us":64,"threads_started":1}
I20260927 03:58:26.356992 18331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:26.352697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41692 (local address 127.15.185.67:36419)
0927 03:58:26.353144 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:26.353148 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:26.353157 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:58:26.353194 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:26.353196 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:26.353211 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:26.353281 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:58:26.353477 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:26.354554 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:26.355816 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:26.356511 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:26.356599 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:26.356658 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:26.356842 (+ 184us) server_negotiation.cc:299] Negotiation successful
0927 03:58:26.356890 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":91,"threads_started":1}
I20260927 03:58:26.357213 18329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:26.352443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.65:45357 (local address 127.0.0.1:35292)
0927 03:58:26.352975 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:26.352981 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:26.353037 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:26.353644 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:26.353645 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:26.353646 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:58:26.353689 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:26.353691 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:26.354529 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:26.354530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:26.356004 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:26.356005 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:26.356975 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:26.356976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:26.357041 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:26.357046 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:58:26.357114 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 03:58:26.357155 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"mutex_wait_us":68,"thread_start_us":100,"threads_started":1}
I20260927 03:58:26.357251 18332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:26.352718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56438 (local address 127.15.185.66:38903)
0927 03:58:26.352862 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:26.352867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:26.352878 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:58:26.352918 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:26.352920 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:26.352936 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:26.353021 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:58:26.353338 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:26.354512 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:26.355997 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:26.356527 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:26.356968 (+ 441us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:26.357048 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:26.357082 (+ 34us) server_negotiation.cc:299] Negotiation successful
0927 03:58:26.357135 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":73,"threads_started":1}
I20260927 03:58:26.357581 18333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:26.352909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35292 (local address 127.15.185.65:45357)
0927 03:58:26.353476 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:26.353480 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:26.353491 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:58:26.353533 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:26.353534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:26.353547 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:26.353647 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:58:26.353747 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:26.354439 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:26.356091 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:26.356821 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:26.356888 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:26.356939 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:26.357414 (+ 475us) server_negotiation.cc:299] Negotiation successful
0927 03:58:26.357464 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":83,"threads_started":1}
I20260927 03:58:26.356927 18330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:26.352443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.67:36419 (local address 127.0.0.1:41692)
0927 03:58:26.352855 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:26.352866 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:26.352920 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:26.353346 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:26.353346 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:26.353348 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:58:26.353403 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:26.353405 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:26.354639 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:26.354640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:26.355725 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:26.355727 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:26.356608 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:26.356609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:26.356688 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:26.356704 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 03:58:26.356777 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 03:58:26.356805 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"mutex_wait_us":52,"thread_start_us":101,"threads_started":1}
I20260927 03:58:26.358276 17965 tablet_service.cc:1467] Processing CreateTablet for tablet 30130e282564440d83f87f863a755dd6 (DEFAULT_TABLE table=test-table [id=15497da3791c489695d7c2783a4cb2c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:58:26.358276 18105 tablet_service.cc:1467] Processing CreateTablet for tablet 30130e282564440d83f87f863a755dd6 (DEFAULT_TABLE table=test-table [id=15497da3791c489695d7c2783a4cb2c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:58:26.358438 18245 tablet_service.cc:1467] Processing CreateTablet for tablet 30130e282564440d83f87f863a755dd6 (DEFAULT_TABLE table=test-table [id=15497da3791c489695d7c2783a4cb2c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:58:26.358641 17965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30130e282564440d83f87f863a755dd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:26.358641 18105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30130e282564440d83f87f863a755dd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:26.358752 18245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30130e282564440d83f87f863a755dd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:26.361348 18334 tablet_bootstrap.cc:492] T 30130e282564440d83f87f863a755dd6 P 1c62cb4ea2bf45a1830e7048550cf4db: Bootstrap starting.
I20260927 03:58:26.362224 18334 tablet_bootstrap.cc:654] T 30130e282564440d83f87f863a755dd6 P 1c62cb4ea2bf45a1830e7048550cf4db: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:26.362468 18336 tablet_bootstrap.cc:492] T 30130e282564440d83f87f863a755dd6 P be26fcebf49a43259cadc0f2b9c0325a: Bootstrap starting.
I20260927 03:58:26.362555 18334 log.cc:824] T 30130e282564440d83f87f863a755dd6 P 1c62cb4ea2bf45a1830e7048550cf4db: Log is configured to *not* fsync() on all Append() calls
I20260927 03:58:26.363095 18336 tablet_bootstrap.cc:654] T 30130e282564440d83f87f863a755dd6 P be26fcebf49a43259cadc0f2b9c0325a: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:26.363224 18335 tablet_bootstrap.cc:492] T 30130e282564440d83f87f863a755dd6 P 47a4ce643e7f40d4b3ff4892217c5352: Bootstrap starting.
I20260927 03:58:26.363374 18334 tablet_bootstrap.cc:492] T 30130e282564440d83f87f863a755dd6 P 1c62cb4ea2bf45a1830e7048550cf4db: No bootstrap required, opened a new log
I20260927 03:58:26.363458 18334 ts_tablet_manager.cc:1397] T 30130e282564440d83f87f863a755dd6 P 1c62cb4ea2bf45a1830e7048550cf4db: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:58:26.363443 18336 log.cc:824] T 30130e282564440d83f87f863a755dd6 P be26fcebf49a43259cadc0f2b9c0325a: Log is configured to *not* fsync() on all Append() calls
I20260927 03:58:26.364023 18335 tablet_bootstrap.cc:654] T 30130e282564440d83f87f863a755dd6 P 47a4ce643e7f40d4b3ff4892217c5352: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:26.364245 18336 tablet_bootstrap.cc:492] T 30130e282564440d83f87f863a755dd6 P be26fcebf49a43259cadc0f2b9c0325a: No bootstrap required, opened a new log
I20260927 03:58:26.364316 18336 ts_tablet_manager.cc:1397] T 30130e282564440d83f87f863a755dd6 P be26fcebf49a43259cadc0f2b9c0325a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:58:26.364400 18335 log.cc:824] T 30130e282564440d83f87f863a755dd6 P 47a4ce643e7f40d4b3ff4892217c5352: Log is configured to *not* fsync() on all Append() calls
I20260927 03:58:26.365068 18335 tablet_bootstrap.cc:492] T 30130e282564440d83f87f863a755dd6 P 47a4ce643e7f40d4b3ff4892217c5352: No bootstrap required, opened a new log
I20260927 03:58:26.365150 18335 ts_tablet_manager.cc:1397] T 30130e282564440d83f87f863a755dd6 P 47a4ce643e7f40d4b3ff4892217c5352: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:58:26.365464 18334 raft_consensus.cc:348] T 30130e282564440d83f87f863a755dd6 P 1c62cb4ea2bf45a1830e7048550cf4db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be26fcebf49a43259cadc0f2b9c0325a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38903 } } peers { permanent_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36419 } } peers { permanent_uuid: "47a4ce643e7f40d4b3ff4892217c5352" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 45357 } }
I20260927 03:58:26.365645 18334 raft_consensus.cc:374] T 30130e282564440d83f87f863a755dd6 P 1c62cb4ea2bf45a1830e7048550cf4db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:26.365681 18334 raft_consensus.cc:729] T 30130e282564440d83f87f863a755dd6 P 1c62cb4ea2bf45a1830e7048550cf4db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c62cb4ea2bf45a1830e7048550cf4db, State: Initialized, Role: FOLLOWER
I20260927 03:58:26.365823 18334 consensus_queue.cc:260] T 30130e282564440d83f87f863a755dd6 P 1c62cb4ea2bf45a1830e7048550cf4db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be26fcebf49a43259cadc0f2b9c0325a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38903 } } peers { permanent_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36419 } } peers { permanent_uuid: "47a4ce643e7f40d4b3ff4892217c5352" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 45357 } }
I20260927 03:58:26.366088 18334 ts_tablet_manager.cc:1428] T 30130e282564440d83f87f863a755dd6 P 1c62cb4ea2bf45a1830e7048550cf4db: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 03:58:26.366196 18311 heartbeater.cc:503] Master 127.15.185.126:41063 was elected leader, sending a full tablet report...
I20260927 03:58:26.366285 18336 raft_consensus.cc:348] T 30130e282564440d83f87f863a755dd6 P be26fcebf49a43259cadc0f2b9c0325a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be26fcebf49a43259cadc0f2b9c0325a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38903 } } peers { permanent_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36419 } } peers { permanent_uuid: "47a4ce643e7f40d4b3ff4892217c5352" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 45357 } }
I20260927 03:58:26.366446 18336 raft_consensus.cc:374] T 30130e282564440d83f87f863a755dd6 P be26fcebf49a43259cadc0f2b9c0325a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:26.366489 18336 raft_consensus.cc:729] T 30130e282564440d83f87f863a755dd6 P be26fcebf49a43259cadc0f2b9c0325a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be26fcebf49a43259cadc0f2b9c0325a, State: Initialized, Role: FOLLOWER
I20260927 03:58:26.366623 18336 consensus_queue.cc:260] T 30130e282564440d83f87f863a755dd6 P be26fcebf49a43259cadc0f2b9c0325a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be26fcebf49a43259cadc0f2b9c0325a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38903 } } peers { permanent_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36419 } } peers { permanent_uuid: "47a4ce643e7f40d4b3ff4892217c5352" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 45357 } }
I20260927 03:58:26.366909 18336 ts_tablet_manager.cc:1428] T 30130e282564440d83f87f863a755dd6 P be26fcebf49a43259cadc0f2b9c0325a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:58:26.366915 18171 heartbeater.cc:503] Master 127.15.185.126:41063 was elected leader, sending a full tablet report...
I20260927 03:58:26.367168 18335 raft_consensus.cc:348] T 30130e282564440d83f87f863a755dd6 P 47a4ce643e7f40d4b3ff4892217c5352 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be26fcebf49a43259cadc0f2b9c0325a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38903 } } peers { permanent_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36419 } } peers { permanent_uuid: "47a4ce643e7f40d4b3ff4892217c5352" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 45357 } }
I20260927 03:58:26.367327 18335 raft_consensus.cc:374] T 30130e282564440d83f87f863a755dd6 P 47a4ce643e7f40d4b3ff4892217c5352 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:26.367376 18335 raft_consensus.cc:729] T 30130e282564440d83f87f863a755dd6 P 47a4ce643e7f40d4b3ff4892217c5352 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47a4ce643e7f40d4b3ff4892217c5352, State: Initialized, Role: FOLLOWER
I20260927 03:58:26.367499 18335 consensus_queue.cc:260] T 30130e282564440d83f87f863a755dd6 P 47a4ce643e7f40d4b3ff4892217c5352 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be26fcebf49a43259cadc0f2b9c0325a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38903 } } peers { permanent_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36419 } } peers { permanent_uuid: "47a4ce643e7f40d4b3ff4892217c5352" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 45357 } }
I20260927 03:58:26.367759 18335 ts_tablet_manager.cc:1428] T 30130e282564440d83f87f863a755dd6 P 47a4ce643e7f40d4b3ff4892217c5352: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:58:26.367834 18031 heartbeater.cc:503] Master 127.15.185.126:41063 was elected leader, sending a full tablet report...
I20260927 03:58:26.375764 18340 raft_consensus.cc:482] T 30130e282564440d83f87f863a755dd6 P 1c62cb4ea2bf45a1830e7048550cf4db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:58:26.375882 18340 raft_consensus.cc:504] T 30130e282564440d83f87f863a755dd6 P 1c62cb4ea2bf45a1830e7048550cf4db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be26fcebf49a43259cadc0f2b9c0325a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38903 } } peers { permanent_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36419 } } peers { permanent_uuid: "47a4ce643e7f40d4b3ff4892217c5352" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 45357 } }
I20260927 03:58:26.376289 18340 leader_election.cc:288] T 30130e282564440d83f87f863a755dd6 P 1c62cb4ea2bf45a1830e7048550cf4db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be26fcebf49a43259cadc0f2b9c0325a (127.15.185.66:38903), 47a4ce643e7f40d4b3ff4892217c5352 (127.15.185.65:45357)
I20260927 03:58:26.379019 18332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:26.376461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:59981 (local address 127.15.185.66:38903)
0927 03:58:26.376505 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:26.376507 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:26.376535 (+ 28us) server_negotiation.cc:407] Connection header received
0927 03:58:26.376644 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:26.376645 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:26.376657 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:26.376714 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:58:26.376977 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:26.377414 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:26.378312 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:26.378692 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:26.378746 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:26.378754 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:26.378914 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 03:58:26.378954 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 03:58:26.379020 18204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:26.376419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.66:38903 (local address 127.15.185.67:59981)
0927 03:58:26.376489 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:26.376501 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:26.376551 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:26.376828 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:26.376829 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:26.376831 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:58:26.376877 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:26.376878 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:26.377493 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:26.377494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:26.378229 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:26.378231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:26.378769 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:26.378770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:26.378848 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:26.378851 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:58:26.378892 (+ 41us) client_negotiation.cc:240] Negotiation successful
0927 03:58:26.378923 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":17}
I20260927 03:58:26.379150 18343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:26.376474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.65:45357 (local address 127.15.185.67:56845)
0927 03:58:26.376643 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:26.376653 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:26.376703 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:26.376800 (+ 97us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:26.376800 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:26.376802 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:58:26.376862 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:26.376863 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:26.377518 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:26.377519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:26.378250 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:26.378251 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:26.378811 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:26.378812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:26.378900 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:26.378906 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 03:58:26.378976 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 03:58:26.379013 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":1,"thread_start_us":110,"threads_started":1}
I20260927 03:58:26.379369 18125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30130e282564440d83f87f863a755dd6" candidate_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be26fcebf49a43259cadc0f2b9c0325a" is_pre_election: true
I20260927 03:58:26.379501 18333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:26.376588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:56845 (local address 127.15.185.65:45357)
0927 03:58:26.376640 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:26.376642 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:26.376710 (+ 68us) server_negotiation.cc:407] Connection header received
0927 03:58:26.376747 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:26.376748 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:26.376755 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:26.376788 (+ 33us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:58:26.376930 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:26.377416 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:26.378359 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:26.378707 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:26.378799 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:26.378807 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:26.379001 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 03:58:26.379038 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 03:58:26.379508 18125 raft_consensus.cc:2454] T 30130e282564440d83f87f863a755dd6 P be26fcebf49a43259cadc0f2b9c0325a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c62cb4ea2bf45a1830e7048550cf4db in term 0.
I20260927 03:58:26.379740 18199 leader_election.cc:302] T 30130e282564440d83f87f863a755dd6 P 1c62cb4ea2bf45a1830e7048550cf4db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c62cb4ea2bf45a1830e7048550cf4db, be26fcebf49a43259cadc0f2b9c0325a; no voters:
I20260927 03:58:26.379778 17982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30130e282564440d83f87f863a755dd6" candidate_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47a4ce643e7f40d4b3ff4892217c5352" is_pre_election: true
I20260927 03:58:26.379894 17982 raft_consensus.cc:2454] T 30130e282564440d83f87f863a755dd6 P 47a4ce643e7f40d4b3ff4892217c5352 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c62cb4ea2bf45a1830e7048550cf4db in term 0.
I20260927 03:58:26.379944 18340 raft_consensus.cc:2781] T 30130e282564440d83f87f863a755dd6 P 1c62cb4ea2bf45a1830e7048550cf4db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:58:26.380019 18340 raft_consensus.cc:482] T 30130e282564440d83f87f863a755dd6 P 1c62cb4ea2bf45a1830e7048550cf4db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:58:26.380076 18340 raft_consensus.cc:3037] T 30130e282564440d83f87f863a755dd6 P 1c62cb4ea2bf45a1830e7048550cf4db [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:26.380756 18340 raft_consensus.cc:504] T 30130e282564440d83f87f863a755dd6 P 1c62cb4ea2bf45a1830e7048550cf4db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be26fcebf49a43259cadc0f2b9c0325a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38903 } } peers { permanent_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36419 } } peers { permanent_uuid: "47a4ce643e7f40d4b3ff4892217c5352" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 45357 } }
I20260927 03:58:26.380893 18340 leader_election.cc:288] T 30130e282564440d83f87f863a755dd6 P 1c62cb4ea2bf45a1830e7048550cf4db [CANDIDATE]: Term 1 election: Requested vote from peers be26fcebf49a43259cadc0f2b9c0325a (127.15.185.66:38903), 47a4ce643e7f40d4b3ff4892217c5352 (127.15.185.65:45357)
I20260927 03:58:26.381038 18125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30130e282564440d83f87f863a755dd6" candidate_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be26fcebf49a43259cadc0f2b9c0325a"
I20260927 03:58:26.381049 17982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30130e282564440d83f87f863a755dd6" candidate_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47a4ce643e7f40d4b3ff4892217c5352"
I20260927 03:58:26.381131 17982 raft_consensus.cc:3037] T 30130e282564440d83f87f863a755dd6 P 47a4ce643e7f40d4b3ff4892217c5352 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:26.381129 18125 raft_consensus.cc:3037] T 30130e282564440d83f87f863a755dd6 P be26fcebf49a43259cadc0f2b9c0325a [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:26.381831 18125 raft_consensus.cc:2454] T 30130e282564440d83f87f863a755dd6 P be26fcebf49a43259cadc0f2b9c0325a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c62cb4ea2bf45a1830e7048550cf4db in term 1.
I20260927 03:58:26.381908 17982 raft_consensus.cc:2454] T 30130e282564440d83f87f863a755dd6 P 47a4ce643e7f40d4b3ff4892217c5352 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c62cb4ea2bf45a1830e7048550cf4db in term 1.
I20260927 03:58:26.381966 18199 leader_election.cc:302] T 30130e282564440d83f87f863a755dd6 P 1c62cb4ea2bf45a1830e7048550cf4db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c62cb4ea2bf45a1830e7048550cf4db, be26fcebf49a43259cadc0f2b9c0325a; no voters:
I20260927 03:58:26.382050 18340 raft_consensus.cc:2781] T 30130e282564440d83f87f863a755dd6 P 1c62cb4ea2bf45a1830e7048550cf4db [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:58:26.382198 18340 raft_consensus.cc:686] T 30130e282564440d83f87f863a755dd6 P 1c62cb4ea2bf45a1830e7048550cf4db [term 1 LEADER]: Becoming Leader. State: Replica: 1c62cb4ea2bf45a1830e7048550cf4db, State: Running, Role: LEADER
I20260927 03:58:26.382294 18340 consensus_queue.cc:237] T 30130e282564440d83f87f863a755dd6 P 1c62cb4ea2bf45a1830e7048550cf4db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be26fcebf49a43259cadc0f2b9c0325a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38903 } } peers { permanent_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36419 } } peers { permanent_uuid: "47a4ce643e7f40d4b3ff4892217c5352" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 45357 } }
I20260927 03:58:26.383113 17828 catalog_manager.cc:5495] T 30130e282564440d83f87f863a755dd6 P 1c62cb4ea2bf45a1830e7048550cf4db reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c62cb4ea2bf45a1830e7048550cf4db (127.15.185.67). New cstate: current_term: 1 leader_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be26fcebf49a43259cadc0f2b9c0325a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36419 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47a4ce643e7f40d4b3ff4892217c5352" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 45357 } health_report { overall_health: UNKNOWN } } }
W20260927 03:58:26.404975 18172 tablet.cc:2369] T 30130e282564440d83f87f863a755dd6 P be26fcebf49a43259cadc0f2b9c0325a: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 03:58:26 dist-test-slave-glgz krb5kdc[17783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481505, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.126@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 03:58:26.422919 17877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:26.415822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36524 (local address 127.15.185.126:41063)
0927 03:58:26.415916 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:26.415918 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:26.415969 (+ 51us) server_negotiation.cc:407] Connection header received
0927 03:58:26.416009 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:26.416010 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:26.416023 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:26.416078 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:26.416510 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:26.417011 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:26.417970 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:26.418206 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:26.418270 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:26.420469 (+ 2199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:26.420470 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:26.420470 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:26.420484 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:26.421971 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:26.422305 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:26.422306 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:26.422306 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:26.422353 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:26.422526 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:26.422526 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:26.422527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:26.422625 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:26.422674 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:26.422788 (+ 114us) server_negotiation.cc:299] Negotiation successful
0927 03:58:26.422854 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 03:58:26.423144 18351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:26.415727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:41063 (local address 127.0.0.1:36524)
0927 03:58:26.415884 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:26.415895 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:26.415963 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:26.416093 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:26.416093 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:26.416395 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:26.416442 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:26.416443 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:26.417122 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:26.417123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:26.417840 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:26.417842 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:26.418296 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:26.418297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:26.418345 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:26.418889 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:26.418893 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:26.420372 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:26.422047 (+ 1675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:26.422053 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:26.422055 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:26.422242 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:26.422409 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:26.422409 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:26.422409 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:26.422464 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:26.422678 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:26.422683 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:26.422722 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 03:58:26.422773 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 03:58:26.422922 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":62,"thread_start_us":72,"threads_started":1}
I20260927 03:58:26.424989 17828 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:36524:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 03:58:26.425225 17828 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:58:26.426746 17965 tablet_service.cc:1467] Processing CreateTablet for tablet 215fd6e7fa64466ca244dd084e6e6984 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2dac6fd47292453c9475ec4add8179ec]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 03:58:26.426783 18105 tablet_service.cc:1467] Processing CreateTablet for tablet 215fd6e7fa64466ca244dd084e6e6984 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2dac6fd47292453c9475ec4add8179ec]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 03:58:26.426793 18245 tablet_service.cc:1467] Processing CreateTablet for tablet 215fd6e7fa64466ca244dd084e6e6984 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2dac6fd47292453c9475ec4add8179ec]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 03:58:26.426916 18105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 215fd6e7fa64466ca244dd084e6e6984. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:26.426916 17965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 215fd6e7fa64466ca244dd084e6e6984. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:26.426951 18245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 215fd6e7fa64466ca244dd084e6e6984. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:26.428064 18334 tablet_bootstrap.cc:492] T 215fd6e7fa64466ca244dd084e6e6984 P 1c62cb4ea2bf45a1830e7048550cf4db: Bootstrap starting.
I20260927 03:58:26.428382 18336 tablet_bootstrap.cc:492] T 215fd6e7fa64466ca244dd084e6e6984 P be26fcebf49a43259cadc0f2b9c0325a: Bootstrap starting.
I20260927 03:58:26.428649 18334 tablet_bootstrap.cc:654] T 215fd6e7fa64466ca244dd084e6e6984 P 1c62cb4ea2bf45a1830e7048550cf4db: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:26.429095 18336 tablet_bootstrap.cc:654] T 215fd6e7fa64466ca244dd084e6e6984 P be26fcebf49a43259cadc0f2b9c0325a: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:26.429425 18334 tablet_bootstrap.cc:492] T 215fd6e7fa64466ca244dd084e6e6984 P 1c62cb4ea2bf45a1830e7048550cf4db: No bootstrap required, opened a new log
I20260927 03:58:26.429469 18334 ts_tablet_manager.cc:1397] T 215fd6e7fa64466ca244dd084e6e6984 P 1c62cb4ea2bf45a1830e7048550cf4db: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 03:58:26.429479 18335 tablet_bootstrap.cc:492] T 215fd6e7fa64466ca244dd084e6e6984 P 47a4ce643e7f40d4b3ff4892217c5352: Bootstrap starting.
I20260927 03:58:26.429618 18334 raft_consensus.cc:348] T 215fd6e7fa64466ca244dd084e6e6984 P 1c62cb4ea2bf45a1830e7048550cf4db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a4ce643e7f40d4b3ff4892217c5352" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 45357 } } peers { permanent_uuid: "be26fcebf49a43259cadc0f2b9c0325a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38903 } } peers { permanent_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36419 } }
I20260927 03:58:26.429692 18334 raft_consensus.cc:374] T 215fd6e7fa64466ca244dd084e6e6984 P 1c62cb4ea2bf45a1830e7048550cf4db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:26.429715 18334 raft_consensus.cc:729] T 215fd6e7fa64466ca244dd084e6e6984 P 1c62cb4ea2bf45a1830e7048550cf4db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c62cb4ea2bf45a1830e7048550cf4db, State: Initialized, Role: FOLLOWER
I20260927 03:58:26.429776 18334 consensus_queue.cc:260] T 215fd6e7fa64466ca244dd084e6e6984 P 1c62cb4ea2bf45a1830e7048550cf4db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a4ce643e7f40d4b3ff4892217c5352" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 45357 } } peers { permanent_uuid: "be26fcebf49a43259cadc0f2b9c0325a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38903 } } peers { permanent_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36419 } }
I20260927 03:58:26.429888 18334 ts_tablet_manager.cc:1428] T 215fd6e7fa64466ca244dd084e6e6984 P 1c62cb4ea2bf45a1830e7048550cf4db: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:26.429949 18340 tablet_replica.cc:439] T 215fd6e7fa64466ca244dd084e6e6984 P 1c62cb4ea2bf45a1830e7048550cf4db: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a4ce643e7f40d4b3ff4892217c5352" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 45357 } } peers { permanent_uuid: "be26fcebf49a43259cadc0f2b9c0325a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38903 } } peers { permanent_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36419 } } }
I20260927 03:58:26.430034 18340 tablet_replica.cc:442] T 215fd6e7fa64466ca244dd084e6e6984 P 1c62cb4ea2bf45a1830e7048550cf4db: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:58:26.430090 18335 tablet_bootstrap.cc:654] T 215fd6e7fa64466ca244dd084e6e6984 P 47a4ce643e7f40d4b3ff4892217c5352: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:26.430678 18336 tablet_bootstrap.cc:492] T 215fd6e7fa64466ca244dd084e6e6984 P be26fcebf49a43259cadc0f2b9c0325a: No bootstrap required, opened a new log
I20260927 03:58:26.430729 18336 ts_tablet_manager.cc:1397] T 215fd6e7fa64466ca244dd084e6e6984 P be26fcebf49a43259cadc0f2b9c0325a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:58:26.430889 18336 raft_consensus.cc:348] T 215fd6e7fa64466ca244dd084e6e6984 P be26fcebf49a43259cadc0f2b9c0325a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a4ce643e7f40d4b3ff4892217c5352" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 45357 } } peers { permanent_uuid: "be26fcebf49a43259cadc0f2b9c0325a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38903 } } peers { permanent_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36419 } }
I20260927 03:58:26.430941 18336 raft_consensus.cc:374] T 215fd6e7fa64466ca244dd084e6e6984 P be26fcebf49a43259cadc0f2b9c0325a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:26.430981 18336 raft_consensus.cc:729] T 215fd6e7fa64466ca244dd084e6e6984 P be26fcebf49a43259cadc0f2b9c0325a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be26fcebf49a43259cadc0f2b9c0325a, State: Initialized, Role: FOLLOWER
I20260927 03:58:26.431030 18336 consensus_queue.cc:260] T 215fd6e7fa64466ca244dd084e6e6984 P be26fcebf49a43259cadc0f2b9c0325a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a4ce643e7f40d4b3ff4892217c5352" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 45357 } } peers { permanent_uuid: "be26fcebf49a43259cadc0f2b9c0325a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38903 } } peers { permanent_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36419 } }
I20260927 03:58:26.431160 18336 ts_tablet_manager.cc:1428] T 215fd6e7fa64466ca244dd084e6e6984 P be26fcebf49a43259cadc0f2b9c0325a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:26.431236 18341 tablet_replica.cc:439] T 215fd6e7fa64466ca244dd084e6e6984 P be26fcebf49a43259cadc0f2b9c0325a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a4ce643e7f40d4b3ff4892217c5352" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 45357 } } peers { permanent_uuid: "be26fcebf49a43259cadc0f2b9c0325a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38903 } } peers { permanent_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36419 } } }
I20260927 03:58:26.431355 18335 tablet_bootstrap.cc:492] T 215fd6e7fa64466ca244dd084e6e6984 P 47a4ce643e7f40d4b3ff4892217c5352: No bootstrap required, opened a new log
I20260927 03:58:26.431348 18341 tablet_replica.cc:442] T 215fd6e7fa64466ca244dd084e6e6984 P be26fcebf49a43259cadc0f2b9c0325a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:58:26.431406 18335 ts_tablet_manager.cc:1397] T 215fd6e7fa64466ca244dd084e6e6984 P 47a4ce643e7f40d4b3ff4892217c5352: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:58:26.431581 18335 raft_consensus.cc:348] T 215fd6e7fa64466ca244dd084e6e6984 P 47a4ce643e7f40d4b3ff4892217c5352 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a4ce643e7f40d4b3ff4892217c5352" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 45357 } } peers { permanent_uuid: "be26fcebf49a43259cadc0f2b9c0325a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38903 } } peers { permanent_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36419 } }
I20260927 03:58:26.431663 18335 raft_consensus.cc:374] T 215fd6e7fa64466ca244dd084e6e6984 P 47a4ce643e7f40d4b3ff4892217c5352 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:26.431690 18335 raft_consensus.cc:729] T 215fd6e7fa64466ca244dd084e6e6984 P 47a4ce643e7f40d4b3ff4892217c5352 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47a4ce643e7f40d4b3ff4892217c5352, State: Initialized, Role: FOLLOWER
I20260927 03:58:26.431737 18335 consensus_queue.cc:260] T 215fd6e7fa64466ca244dd084e6e6984 P 47a4ce643e7f40d4b3ff4892217c5352 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a4ce643e7f40d4b3ff4892217c5352" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 45357 } } peers { permanent_uuid: "be26fcebf49a43259cadc0f2b9c0325a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38903 } } peers { permanent_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36419 } }
I20260927 03:58:26.431849 18335 ts_tablet_manager.cc:1428] T 215fd6e7fa64466ca244dd084e6e6984 P 47a4ce643e7f40d4b3ff4892217c5352: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:26.431901 18342 tablet_replica.cc:439] T 215fd6e7fa64466ca244dd084e6e6984 P 47a4ce643e7f40d4b3ff4892217c5352: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a4ce643e7f40d4b3ff4892217c5352" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 45357 } } peers { permanent_uuid: "be26fcebf49a43259cadc0f2b9c0325a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38903 } } peers { permanent_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36419 } } }
I20260927 03:58:26.431994 18342 tablet_replica.cc:442] T 215fd6e7fa64466ca244dd084e6e6984 P 47a4ce643e7f40d4b3ff4892217c5352: This TxnStatusTablet replica's current role is: FOLLOWER
W20260927 03:58:26.490491 18032 tablet.cc:2369] T 215fd6e7fa64466ca244dd084e6e6984 P 47a4ce643e7f40d4b3ff4892217c5352: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 03:58:26.566525 18312 tablet.cc:2369] T 215fd6e7fa64466ca244dd084e6e6984 P 1c62cb4ea2bf45a1830e7048550cf4db: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:58:26.678745 18341 raft_consensus.cc:482] T 215fd6e7fa64466ca244dd084e6e6984 P be26fcebf49a43259cadc0f2b9c0325a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:58:26.678848 18341 raft_consensus.cc:504] T 215fd6e7fa64466ca244dd084e6e6984 P be26fcebf49a43259cadc0f2b9c0325a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a4ce643e7f40d4b3ff4892217c5352" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 45357 } } peers { permanent_uuid: "be26fcebf49a43259cadc0f2b9c0325a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38903 } } peers { permanent_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36419 } }
I20260927 03:58:26.679294 18341 leader_election.cc:288] T 215fd6e7fa64466ca244dd084e6e6984 P be26fcebf49a43259cadc0f2b9c0325a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47a4ce643e7f40d4b3ff4892217c5352 (127.15.185.65:45357), 1c62cb4ea2bf45a1830e7048550cf4db (127.15.185.67:36419)
I20260927 03:58:26.682317 18333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:26.679584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:48847 (local address 127.15.185.65:45357)
0927 03:58:26.679633 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:26.679636 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:26.679746 (+ 110us) server_negotiation.cc:407] Connection header received
0927 03:58:26.679820 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:26.679822 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:26.679831 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:26.679919 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:58:26.680060 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:26.680534 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:26.681560 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:26.681955 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:26.682040 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:26.682048 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:26.682212 (+ 164us) server_negotiation.cc:299] Negotiation successful
0927 03:58:26.682248 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 03:58:26.682313 18331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:26.679637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:49443 (local address 127.15.185.67:36419)
0927 03:58:26.679691 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:26.679694 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:26.679764 (+ 70us) server_negotiation.cc:407] Connection header received
0927 03:58:26.679848 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:26.679849 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:26.679858 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:26.679941 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:58:26.680149 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:26.680614 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:26.681563 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:26.681940 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:26.682025 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:26.682034 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:26.682198 (+ 164us) server_negotiation.cc:299] Negotiation successful
0927 03:58:26.682234 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 03:58:26.682335 18355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:26.679557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.67:36419 (local address 127.15.185.66:49443)
0927 03:58:26.679687 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:26.679698 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:26.679781 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:26.679950 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:26.679950 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:26.679952 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:58:26.680064 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:26.680066 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:26.680699 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:26.680700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:26.681473 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:26.681475 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:26.682035 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:26.682036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:26.682113 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:26.682117 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:58:26.682191 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 03:58:26.682234 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":57,"threads_started":1}
I20260927 03:58:26.682344 18356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:26.679499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.65:45357 (local address 127.15.185.66:48847)
0927 03:58:26.679655 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:26.679666 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:26.679760 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:26.679926 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:26.679926 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:26.679929 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:58:26.679983 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:26.679985 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:26.680640 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:26.680641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:26.681459 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:26.681462 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:26.682038 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:26.682039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:26.682124 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:26.682128 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:58:26.682204 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 03:58:26.682248 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":11,"thread_start_us":76,"threads_started":1}
I20260927 03:58:26.682644 18265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "215fd6e7fa64466ca244dd084e6e6984" candidate_uuid: "be26fcebf49a43259cadc0f2b9c0325a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" is_pre_election: true
I20260927 03:58:26.682647 17982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "215fd6e7fa64466ca244dd084e6e6984" candidate_uuid: "be26fcebf49a43259cadc0f2b9c0325a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47a4ce643e7f40d4b3ff4892217c5352" is_pre_election: true
I20260927 03:58:26.682749 17982 raft_consensus.cc:2454] T 215fd6e7fa64466ca244dd084e6e6984 P 47a4ce643e7f40d4b3ff4892217c5352 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be26fcebf49a43259cadc0f2b9c0325a in term 0.
I20260927 03:58:26.682760 18265 raft_consensus.cc:2454] T 215fd6e7fa64466ca244dd084e6e6984 P 1c62cb4ea2bf45a1830e7048550cf4db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be26fcebf49a43259cadc0f2b9c0325a in term 0.
I20260927 03:58:26.682912 18058 leader_election.cc:302] T 215fd6e7fa64466ca244dd084e6e6984 P be26fcebf49a43259cadc0f2b9c0325a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47a4ce643e7f40d4b3ff4892217c5352, be26fcebf49a43259cadc0f2b9c0325a; no voters:
I20260927 03:58:26.683123 18341 raft_consensus.cc:2781] T 215fd6e7fa64466ca244dd084e6e6984 P be26fcebf49a43259cadc0f2b9c0325a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:58:26.683171 18341 raft_consensus.cc:482] T 215fd6e7fa64466ca244dd084e6e6984 P be26fcebf49a43259cadc0f2b9c0325a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:58:26.683193 18341 raft_consensus.cc:3037] T 215fd6e7fa64466ca244dd084e6e6984 P be26fcebf49a43259cadc0f2b9c0325a [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:26.683923 18341 raft_consensus.cc:504] T 215fd6e7fa64466ca244dd084e6e6984 P be26fcebf49a43259cadc0f2b9c0325a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a4ce643e7f40d4b3ff4892217c5352" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 45357 } } peers { permanent_uuid: "be26fcebf49a43259cadc0f2b9c0325a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38903 } } peers { permanent_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36419 } }
I20260927 03:58:26.684105 18341 leader_election.cc:288] T 215fd6e7fa64466ca244dd084e6e6984 P be26fcebf49a43259cadc0f2b9c0325a [CANDIDATE]: Term 1 election: Requested vote from peers 47a4ce643e7f40d4b3ff4892217c5352 (127.15.185.65:45357), 1c62cb4ea2bf45a1830e7048550cf4db (127.15.185.67:36419)
I20260927 03:58:26.684440 17982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "215fd6e7fa64466ca244dd084e6e6984" candidate_uuid: "be26fcebf49a43259cadc0f2b9c0325a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47a4ce643e7f40d4b3ff4892217c5352"
I20260927 03:58:26.684475 18265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "215fd6e7fa64466ca244dd084e6e6984" candidate_uuid: "be26fcebf49a43259cadc0f2b9c0325a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c62cb4ea2bf45a1830e7048550cf4db"
I20260927 03:58:26.684551 18265 raft_consensus.cc:3037] T 215fd6e7fa64466ca244dd084e6e6984 P 1c62cb4ea2bf45a1830e7048550cf4db [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:26.684554 17982 raft_consensus.cc:3037] T 215fd6e7fa64466ca244dd084e6e6984 P 47a4ce643e7f40d4b3ff4892217c5352 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:26.685153 17982 raft_consensus.cc:2454] T 215fd6e7fa64466ca244dd084e6e6984 P 47a4ce643e7f40d4b3ff4892217c5352 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be26fcebf49a43259cadc0f2b9c0325a in term 1.
I20260927 03:58:26.685178 18265 raft_consensus.cc:2454] T 215fd6e7fa64466ca244dd084e6e6984 P 1c62cb4ea2bf45a1830e7048550cf4db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be26fcebf49a43259cadc0f2b9c0325a in term 1.
I20260927 03:58:26.685364 18056 leader_election.cc:302] T 215fd6e7fa64466ca244dd084e6e6984 P be26fcebf49a43259cadc0f2b9c0325a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c62cb4ea2bf45a1830e7048550cf4db, be26fcebf49a43259cadc0f2b9c0325a; no voters:
I20260927 03:58:26.685456 18341 raft_consensus.cc:2781] T 215fd6e7fa64466ca244dd084e6e6984 P be26fcebf49a43259cadc0f2b9c0325a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:58:26.685586 18341 raft_consensus.cc:686] T 215fd6e7fa64466ca244dd084e6e6984 P be26fcebf49a43259cadc0f2b9c0325a [term 1 LEADER]: Becoming Leader. State: Replica: be26fcebf49a43259cadc0f2b9c0325a, State: Running, Role: LEADER
I20260927 03:58:26.685700 18341 consensus_queue.cc:237] T 215fd6e7fa64466ca244dd084e6e6984 P be26fcebf49a43259cadc0f2b9c0325a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a4ce643e7f40d4b3ff4892217c5352" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 45357 } } peers { permanent_uuid: "be26fcebf49a43259cadc0f2b9c0325a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38903 } } peers { permanent_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36419 } }
I20260927 03:58:26.686193 18357 tablet_replica.cc:439] T 215fd6e7fa64466ca244dd084e6e6984 P be26fcebf49a43259cadc0f2b9c0325a: TxnStatusTablet state changed. Reason: New leader be26fcebf49a43259cadc0f2b9c0325a. Latest consensus state: current_term: 1 leader_uuid: "be26fcebf49a43259cadc0f2b9c0325a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a4ce643e7f40d4b3ff4892217c5352" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 45357 } } peers { permanent_uuid: "be26fcebf49a43259cadc0f2b9c0325a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38903 } } peers { permanent_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36419 } } }
I20260927 03:58:26.686321 18357 tablet_replica.cc:442] T 215fd6e7fa64466ca244dd084e6e6984 P be26fcebf49a43259cadc0f2b9c0325a: This TxnStatusTablet replica's current role is: LEADER
I20260927 03:58:26.686512 18359 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 03:58:26.686617 18359 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 03:58:26.686559 17828 catalog_manager.cc:5495] T 215fd6e7fa64466ca244dd084e6e6984 P be26fcebf49a43259cadc0f2b9c0325a reported cstate change: term changed from 0 to 1, leader changed from <none> to be26fcebf49a43259cadc0f2b9c0325a (127.15.185.66). New cstate: current_term: 1 leader_uuid: "be26fcebf49a43259cadc0f2b9c0325a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a4ce643e7f40d4b3ff4892217c5352" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 45357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be26fcebf49a43259cadc0f2b9c0325a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36419 } health_report { overall_health: UNKNOWN } } }
I20260927 03:58:26.715214 18351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:26.712267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.66:38903 (local address 127.0.0.1:56450)
0927 03:58:26.712341 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:26.712354 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:26.712426 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:26.712606 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:26.712607 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:26.712608 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 03:58:26.712662 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:26.712664 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:26.713485 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:26.713486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:26.714261 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:26.714263 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:26.714635 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:26.714636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:26.714703 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:26.714726 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 03:58:26.715004 (+ 278us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 03:58:26.715013 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 03:58:26.715090 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 03:58:26.715138 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 03:58:26.715214 18332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:26.712346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56450 (local address 127.15.185.66:38903)
0927 03:58:26.712409 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:26.712412 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:26.712421 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:58:26.712502 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:26.712503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:26.712522 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:26.712604 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:58:26.712739 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:26.713400 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:26.714349 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:26.714549 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:26.714635 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:26.714821 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:58:26.714934 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:58:26.715030 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:26.715100 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 03:58:26.715122 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 03:58:26.717096 17982 raft_consensus.cc:1261] T 215fd6e7fa64466ca244dd084e6e6984 P 47a4ce643e7f40d4b3ff4892217c5352 [term 1 FOLLOWER]: Refusing update from remote peer be26fcebf49a43259cadc0f2b9c0325a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:58:26.717095 18265 raft_consensus.cc:1261] T 215fd6e7fa64466ca244dd084e6e6984 P 1c62cb4ea2bf45a1830e7048550cf4db [term 1 FOLLOWER]: Refusing update from remote peer be26fcebf49a43259cadc0f2b9c0325a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:58:26.717455 18357 consensus_queue.cc:1035] T 215fd6e7fa64466ca244dd084e6e6984 P be26fcebf49a43259cadc0f2b9c0325a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:58:26.717612 18341 consensus_queue.cc:1035] T 215fd6e7fa64466ca244dd084e6e6984 P be26fcebf49a43259cadc0f2b9c0325a [LEADER]: Connected to new peer: Peer: permanent_uuid: "47a4ce643e7f40d4b3ff4892217c5352" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 45357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:58:26.719004 18342 tablet_replica.cc:439] T 215fd6e7fa64466ca244dd084e6e6984 P 47a4ce643e7f40d4b3ff4892217c5352: TxnStatusTablet state changed. Reason: New leader be26fcebf49a43259cadc0f2b9c0325a. Latest consensus state: current_term: 1 leader_uuid: "be26fcebf49a43259cadc0f2b9c0325a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a4ce643e7f40d4b3ff4892217c5352" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 45357 } } peers { permanent_uuid: "be26fcebf49a43259cadc0f2b9c0325a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38903 } } peers { permanent_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36419 } } }
I20260927 03:58:26.719123 18342 tablet_replica.cc:442] T 215fd6e7fa64466ca244dd084e6e6984 P 47a4ce643e7f40d4b3ff4892217c5352: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:58:26.719242 18340 tablet_replica.cc:439] T 215fd6e7fa64466ca244dd084e6e6984 P 1c62cb4ea2bf45a1830e7048550cf4db: TxnStatusTablet state changed. Reason: New leader be26fcebf49a43259cadc0f2b9c0325a. Latest consensus state: current_term: 1 leader_uuid: "be26fcebf49a43259cadc0f2b9c0325a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a4ce643e7f40d4b3ff4892217c5352" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 45357 } } peers { permanent_uuid: "be26fcebf49a43259cadc0f2b9c0325a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38903 } } peers { permanent_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36419 } } }
I20260927 03:58:26.719336 18340 tablet_replica.cc:442] T 215fd6e7fa64466ca244dd084e6e6984 P 1c62cb4ea2bf45a1830e7048550cf4db: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:58:26.719681 18362 mvcc.cc:205] Tried to move back new op lower bound from 7333812251510661120 to 7333812251385389056. Current Snapshot: MvccSnapshot[applied={T|T < 7333812251510661120}]
I20260927 03:58:26.719775 18357 tablet_replica.cc:439] T 215fd6e7fa64466ca244dd084e6e6984 P be26fcebf49a43259cadc0f2b9c0325a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "be26fcebf49a43259cadc0f2b9c0325a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a4ce643e7f40d4b3ff4892217c5352" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 45357 } } peers { permanent_uuid: "be26fcebf49a43259cadc0f2b9c0325a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38903 } } peers { permanent_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36419 } } }
I20260927 03:58:26.719846 18357 tablet_replica.cc:442] T 215fd6e7fa64466ca244dd084e6e6984 P be26fcebf49a43259cadc0f2b9c0325a: This TxnStatusTablet replica's current role is: LEADER
I20260927 03:58:26.720428 18340 tablet_replica.cc:439] T 215fd6e7fa64466ca244dd084e6e6984 P 1c62cb4ea2bf45a1830e7048550cf4db: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "be26fcebf49a43259cadc0f2b9c0325a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a4ce643e7f40d4b3ff4892217c5352" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 45357 } } peers { permanent_uuid: "be26fcebf49a43259cadc0f2b9c0325a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38903 } } peers { permanent_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36419 } } }
I20260927 03:58:26.720508 18340 tablet_replica.cc:442] T 215fd6e7fa64466ca244dd084e6e6984 P 1c62cb4ea2bf45a1830e7048550cf4db: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:58:26.721004 18357 tablet_replica.cc:439] T 215fd6e7fa64466ca244dd084e6e6984 P be26fcebf49a43259cadc0f2b9c0325a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "be26fcebf49a43259cadc0f2b9c0325a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a4ce643e7f40d4b3ff4892217c5352" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 45357 } } peers { permanent_uuid: "be26fcebf49a43259cadc0f2b9c0325a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38903 } } peers { permanent_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36419 } } }
I20260927 03:58:26.721079 18357 tablet_replica.cc:442] T 215fd6e7fa64466ca244dd084e6e6984 P be26fcebf49a43259cadc0f2b9c0325a: This TxnStatusTablet replica's current role is: LEADER
I20260927 03:58:26.721747 18342 tablet_replica.cc:439] T 215fd6e7fa64466ca244dd084e6e6984 P 47a4ce643e7f40d4b3ff4892217c5352: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "be26fcebf49a43259cadc0f2b9c0325a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a4ce643e7f40d4b3ff4892217c5352" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 45357 } } peers { permanent_uuid: "be26fcebf49a43259cadc0f2b9c0325a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38903 } } peers { permanent_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36419 } } }
I20260927 03:58:26.721827 18342 tablet_replica.cc:442] T 215fd6e7fa64466ca244dd084e6e6984 P 47a4ce643e7f40d4b3ff4892217c5352: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:58:26.722213 18365 mvcc.cc:205] Tried to move back new op lower bound from 7333812251510661120 to 7333812251385389056. Current Snapshot: MvccSnapshot[applied={T|T < 7333812251510661120 or (T in {7333812251510661120})}]
I20260927 03:58:26.725059 18331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:26.722440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41704 (local address 127.15.185.67:36419)
0927 03:58:26.722496 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:26.722498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:26.722507 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:58:26.722536 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:26.722537 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:26.722554 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:26.722629 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:58:26.722737 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:26.723513 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:26.724258 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:26.724557 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:26.724626 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:26.724750 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:58:26.724865 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:58:26.724938 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:26.724969 (+ 31us) server_negotiation.cc:299] Negotiation successful
0927 03:58:26.724990 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 03:58:26.729693 18317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:26.727415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.66:38903 (local address 127.15.185.67:57575)
0927 03:58:26.727483 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:26.727494 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:26.727534 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:26.727669 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:26.727670 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:26.727671 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 03:58:26.727724 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:26.727725 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:26.728320 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:26.728321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:26.728875 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:26.728876 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:26.729242 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:26.729243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:26.729306 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:26.729322 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 03:58:26.729530 (+ 208us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 03:58:26.729532 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 03:58:26.729592 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 03:58:26.729631 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 03:58:26.729686 18332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:26.727478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:57575 (local address 127.15.185.66:38903)
0927 03:58:26.727522 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:26.727524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:26.727534 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:58:26.727603 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:26.727604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:26.727614 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:26.727667 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:58:26.727786 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:26.728239 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:26.728959 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:26.729171 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:26.729244 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:26.729430 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:58:26.729461 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:58:26.729531 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:26.729596 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 03:58:26.729616 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 03:58:26.732026 17982 raft_consensus.cc:1261] T 30130e282564440d83f87f863a755dd6 P 47a4ce643e7f40d4b3ff4892217c5352 [term 1 FOLLOWER]: Refusing update from remote peer 1c62cb4ea2bf45a1830e7048550cf4db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:58:26.732115 18125 raft_consensus.cc:1261] T 30130e282564440d83f87f863a755dd6 P be26fcebf49a43259cadc0f2b9c0325a [term 1 FOLLOWER]: Refusing update from remote peer 1c62cb4ea2bf45a1830e7048550cf4db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:58:26.732314 18344 consensus_queue.cc:1035] T 30130e282564440d83f87f863a755dd6 P 1c62cb4ea2bf45a1830e7048550cf4db [LEADER]: Connected to new peer: Peer: permanent_uuid: "47a4ce643e7f40d4b3ff4892217c5352" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 45357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:58:26.732445 18340 consensus_queue.cc:1035] T 30130e282564440d83f87f863a755dd6 P 1c62cb4ea2bf45a1830e7048550cf4db [LEADER]: Connected to new peer: Peer: permanent_uuid: "be26fcebf49a43259cadc0f2b9c0325a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:58:26.733713 18363 mvcc.cc:205] Tried to move back new op lower bound from 7333812251572416512 to 7333812250142490624. Current Snapshot: MvccSnapshot[applied={T|T < 7333812251572416512}]
I20260927 03:58:26.740989 18331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:26.738238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:49223 (local address 127.15.185.67:36419)
0927 03:58:26.738282 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:26.738284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:26.738400 (+ 116us) server_negotiation.cc:407] Connection header received
0927 03:58:26.738489 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:26.738490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:26.738501 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:26.738556 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:58:26.738697 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:26.739238 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:26.740337 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:26.740536 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:26.740592 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:26.740767 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:58:26.740791 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:58:26.740823 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:26.740875 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 03:58:26.740895 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 03:58:26.741009 18378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:26.738201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.67:36419 (local address 127.15.185.66:49223)
0927 03:58:26.738337 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:26.738349 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:26.738411 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:26.738578 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:26.738579 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:26.738580 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 03:58:26.738641 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:26.738642 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:26.739292 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:26.739293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:26.740255 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:26.740258 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:26.740608 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:26.740609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:26.740677 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:26.740692 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 03:58:26.740824 (+ 132us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 03:58:26.740826 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 03:58:26.740868 (+ 42us) client_negotiation.cc:240] Negotiation successful
0927 03:58:26.740910 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":78,"threads_started":1}
I20260927 03:58:26.750289 17827 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36520:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:58:26.750414 17827 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36520:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:58:26.751225 17827 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 67a6d9bd1a4e47818b720084e11ccd8a: Sending DeleteTablet for 3 replicas of tablet 30130e282564440d83f87f863a755dd6
I20260927 03:58:26.751540 18245 tablet_service.cc:1514] Processing DeleteTablet for tablet 30130e282564440d83f87f863a755dd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:58:26 UTC) from {username='kudu', principal='kudu/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:41692
I20260927 03:58:26.751579 17965 tablet_service.cc:1514] Processing DeleteTablet for tablet 30130e282564440d83f87f863a755dd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:58:26 UTC) from {username='kudu', principal='kudu/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:35292
I20260927 03:58:26.751842 18105 tablet_service.cc:1514] Processing DeleteTablet for tablet 30130e282564440d83f87f863a755dd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:58:26 UTC) from {username='kudu', principal='kudu/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:56438
I20260927 03:58:26.752108 18384 tablet_replica.cc:330] T 30130e282564440d83f87f863a755dd6 P be26fcebf49a43259cadc0f2b9c0325a: stopping tablet replica
I20260927 03:58:26.752086 17827 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36520:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:58:26.752238 17827 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:58:26.752241 18384 raft_consensus.cc:2229] T 30130e282564440d83f87f863a755dd6 P be26fcebf49a43259cadc0f2b9c0325a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:58:26.752297 18384 raft_consensus.cc:2258] T 30130e282564440d83f87f863a755dd6 P be26fcebf49a43259cadc0f2b9c0325a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:58:26.752434 18383 tablet_replica.cc:330] T 30130e282564440d83f87f863a755dd6 P 47a4ce643e7f40d4b3ff4892217c5352: stopping tablet replica
I20260927 03:58:26.752564 18383 raft_consensus.cc:2229] T 30130e282564440d83f87f863a755dd6 P 47a4ce643e7f40d4b3ff4892217c5352 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:58:26.752678 18383 raft_consensus.cc:2258] T 30130e282564440d83f87f863a755dd6 P 47a4ce643e7f40d4b3ff4892217c5352 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:58:26.752743 18384 ts_tablet_manager.cc:1905] T 30130e282564440d83f87f863a755dd6 P be26fcebf49a43259cadc0f2b9c0325a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:58:26.753286 18382 tablet_replica.cc:330] T 30130e282564440d83f87f863a755dd6 P 1c62cb4ea2bf45a1830e7048550cf4db: stopping tablet replica
I20260927 03:58:26.753425 18382 raft_consensus.cc:2229] T 30130e282564440d83f87f863a755dd6 P 1c62cb4ea2bf45a1830e7048550cf4db [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:58:26.753562 18382 raft_consensus.cc:2258] T 30130e282564440d83f87f863a755dd6 P 1c62cb4ea2bf45a1830e7048550cf4db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:58:26.753651 17965 tablet_service.cc:1467] Processing CreateTablet for tablet 9c962e4019054cffb50c200cb2fedf5f (DEFAULT_TABLE table=test-table [id=2703ab1b1ef94908852b9234d0a44a82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:58:26.753770 17965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c962e4019054cffb50c200cb2fedf5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:26.754037 18382 ts_tablet_manager.cc:1905] T 30130e282564440d83f87f863a755dd6 P 1c62cb4ea2bf45a1830e7048550cf4db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:58:26.755167 18335 tablet_bootstrap.cc:492] T 9c962e4019054cffb50c200cb2fedf5f P 47a4ce643e7f40d4b3ff4892217c5352: Bootstrap starting.
I20260927 03:58:26.755200 18105 tablet_service.cc:1467] Processing CreateTablet for tablet 9c962e4019054cffb50c200cb2fedf5f (DEFAULT_TABLE table=test-table [id=2703ab1b1ef94908852b9234d0a44a82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:58:26.755332 18245 tablet_service.cc:1467] Processing CreateTablet for tablet 9c962e4019054cffb50c200cb2fedf5f (DEFAULT_TABLE table=test-table [id=2703ab1b1ef94908852b9234d0a44a82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:58:26.755421 18245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c962e4019054cffb50c200cb2fedf5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:26.755616 18335 tablet_bootstrap.cc:654] T 9c962e4019054cffb50c200cb2fedf5f P 47a4ce643e7f40d4b3ff4892217c5352: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:26.756001 18382 ts_tablet_manager.cc:1918] T 30130e282564440d83f87f863a755dd6 P 1c62cb4ea2bf45a1830e7048550cf4db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:58:26.756072 18382 log.cc:1196] T 30130e282564440d83f87f863a755dd6 P 1c62cb4ea2bf45a1830e7048550cf4db: Deleting WAL directory at /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/ts-2/wal/wals/30130e282564440d83f87f863a755dd6
I20260927 03:58:26.756310 18382 ts_tablet_manager.cc:1939] T 30130e282564440d83f87f863a755dd6 P 1c62cb4ea2bf45a1830e7048550cf4db: Deleting consensus metadata
I20260927 03:58:26.756736 18335 tablet_bootstrap.cc:492] T 9c962e4019054cffb50c200cb2fedf5f P 47a4ce643e7f40d4b3ff4892217c5352: No bootstrap required, opened a new log
I20260927 03:58:26.756783 18335 ts_tablet_manager.cc:1397] T 9c962e4019054cffb50c200cb2fedf5f P 47a4ce643e7f40d4b3ff4892217c5352: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:58:26.756852 18105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c962e4019054cffb50c200cb2fedf5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:26.756916 18335 raft_consensus.cc:348] T 9c962e4019054cffb50c200cb2fedf5f P 47a4ce643e7f40d4b3ff4892217c5352 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a4ce643e7f40d4b3ff4892217c5352" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 45357 } } peers { permanent_uuid: "be26fcebf49a43259cadc0f2b9c0325a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38903 } } peers { permanent_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36419 } }
I20260927 03:58:26.756990 18335 raft_consensus.cc:374] T 9c962e4019054cffb50c200cb2fedf5f P 47a4ce643e7f40d4b3ff4892217c5352 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:26.757009 18335 raft_consensus.cc:729] T 9c962e4019054cffb50c200cb2fedf5f P 47a4ce643e7f40d4b3ff4892217c5352 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47a4ce643e7f40d4b3ff4892217c5352, State: Initialized, Role: FOLLOWER
I20260927 03:58:26.757040 17812 catalog_manager.cc:4841] TS 1c62cb4ea2bf45a1830e7048550cf4db (127.15.185.67:36419): tablet 30130e282564440d83f87f863a755dd6 (table test-table [id=15497da3791c489695d7c2783a4cb2c5]) successfully deleted
I20260927 03:58:26.757050 18335 consensus_queue.cc:260] T 9c962e4019054cffb50c200cb2fedf5f P 47a4ce643e7f40d4b3ff4892217c5352 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a4ce643e7f40d4b3ff4892217c5352" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 45357 } } peers { permanent_uuid: "be26fcebf49a43259cadc0f2b9c0325a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38903 } } peers { permanent_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36419 } }
I20260927 03:58:26.757128 18335 ts_tablet_manager.cc:1428] T 9c962e4019054cffb50c200cb2fedf5f P 47a4ce643e7f40d4b3ff4892217c5352: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:26.757210 18334 tablet_bootstrap.cc:492] T 9c962e4019054cffb50c200cb2fedf5f P 1c62cb4ea2bf45a1830e7048550cf4db: Bootstrap starting.
I20260927 03:58:26.757462 18383 ts_tablet_manager.cc:1905] T 30130e282564440d83f87f863a755dd6 P 47a4ce643e7f40d4b3ff4892217c5352: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:58:26.757584 18334 tablet_bootstrap.cc:654] T 9c962e4019054cffb50c200cb2fedf5f P 1c62cb4ea2bf45a1830e7048550cf4db: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:26.758267 18336 tablet_bootstrap.cc:492] T 9c962e4019054cffb50c200cb2fedf5f P be26fcebf49a43259cadc0f2b9c0325a: Bootstrap starting.
I20260927 03:58:26.758541 18336 tablet_bootstrap.cc:654] T 9c962e4019054cffb50c200cb2fedf5f P be26fcebf49a43259cadc0f2b9c0325a: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:26.758823 18334 tablet_bootstrap.cc:492] T 9c962e4019054cffb50c200cb2fedf5f P 1c62cb4ea2bf45a1830e7048550cf4db: No bootstrap required, opened a new log
I20260927 03:58:26.758865 18334 ts_tablet_manager.cc:1397] T 9c962e4019054cffb50c200cb2fedf5f P 1c62cb4ea2bf45a1830e7048550cf4db: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 03:58:26.759017 18384 ts_tablet_manager.cc:1918] T 30130e282564440d83f87f863a755dd6 P be26fcebf49a43259cadc0f2b9c0325a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:58:26.759017 18334 raft_consensus.cc:348] T 9c962e4019054cffb50c200cb2fedf5f P 1c62cb4ea2bf45a1830e7048550cf4db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a4ce643e7f40d4b3ff4892217c5352" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 45357 } } peers { permanent_uuid: "be26fcebf49a43259cadc0f2b9c0325a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38903 } } peers { permanent_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36419 } }
I20260927 03:58:26.759114 18334 raft_consensus.cc:374] T 9c962e4019054cffb50c200cb2fedf5f P 1c62cb4ea2bf45a1830e7048550cf4db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:26.759140 18334 raft_consensus.cc:729] T 9c962e4019054cffb50c200cb2fedf5f P 1c62cb4ea2bf45a1830e7048550cf4db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c62cb4ea2bf45a1830e7048550cf4db, State: Initialized, Role: FOLLOWER
I20260927 03:58:26.759124 18384 log.cc:1196] T 30130e282564440d83f87f863a755dd6 P be26fcebf49a43259cadc0f2b9c0325a: Deleting WAL directory at /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/ts-1/wal/wals/30130e282564440d83f87f863a755dd6
I20260927 03:58:26.759181 18334 consensus_queue.cc:260] T 9c962e4019054cffb50c200cb2fedf5f P 1c62cb4ea2bf45a1830e7048550cf4db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a4ce643e7f40d4b3ff4892217c5352" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 45357 } } peers { permanent_uuid: "be26fcebf49a43259cadc0f2b9c0325a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38903 } } peers { permanent_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36419 } }
I20260927 03:58:26.759212 18383 ts_tablet_manager.cc:1918] T 30130e282564440d83f87f863a755dd6 P 47a4ce643e7f40d4b3ff4892217c5352: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:58:26.759260 18334 ts_tablet_manager.cc:1428] T 9c962e4019054cffb50c200cb2fedf5f P 1c62cb4ea2bf45a1830e7048550cf4db: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260927 03:58:26.759270 18383 log.cc:1196] T 30130e282564440d83f87f863a755dd6 P 47a4ce643e7f40d4b3ff4892217c5352: Deleting WAL directory at /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/ts-0/wal/wals/30130e282564440d83f87f863a755dd6
I20260927 03:58:26.759415 18384 ts_tablet_manager.cc:1939] T 30130e282564440d83f87f863a755dd6 P be26fcebf49a43259cadc0f2b9c0325a: Deleting consensus metadata
I20260927 03:58:26.759502 18383 ts_tablet_manager.cc:1939] T 30130e282564440d83f87f863a755dd6 P 47a4ce643e7f40d4b3ff4892217c5352: Deleting consensus metadata
I20260927 03:58:26.759796 17815 catalog_manager.cc:4841] TS be26fcebf49a43259cadc0f2b9c0325a (127.15.185.66:38903): tablet 30130e282564440d83f87f863a755dd6 (table test-table [id=15497da3791c489695d7c2783a4cb2c5]) successfully deleted
I20260927 03:58:26.760106 17814 catalog_manager.cc:4841] TS 47a4ce643e7f40d4b3ff4892217c5352 (127.15.185.65:45357): tablet 30130e282564440d83f87f863a755dd6 (table test-table [id=15497da3791c489695d7c2783a4cb2c5]) successfully deleted
I20260927 03:58:26.763427 18336 tablet_bootstrap.cc:492] T 9c962e4019054cffb50c200cb2fedf5f P be26fcebf49a43259cadc0f2b9c0325a: No bootstrap required, opened a new log
I20260927 03:58:26.763504 18336 ts_tablet_manager.cc:1397] T 9c962e4019054cffb50c200cb2fedf5f P be26fcebf49a43259cadc0f2b9c0325a: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260927 03:58:26.763849 18336 raft_consensus.cc:348] T 9c962e4019054cffb50c200cb2fedf5f P be26fcebf49a43259cadc0f2b9c0325a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a4ce643e7f40d4b3ff4892217c5352" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 45357 } } peers { permanent_uuid: "be26fcebf49a43259cadc0f2b9c0325a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38903 } } peers { permanent_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36419 } }
I20260927 03:58:26.763932 18336 raft_consensus.cc:374] T 9c962e4019054cffb50c200cb2fedf5f P be26fcebf49a43259cadc0f2b9c0325a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:26.763958 18336 raft_consensus.cc:729] T 9c962e4019054cffb50c200cb2fedf5f P be26fcebf49a43259cadc0f2b9c0325a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be26fcebf49a43259cadc0f2b9c0325a, State: Initialized, Role: FOLLOWER
I20260927 03:58:26.764011 18336 consensus_queue.cc:260] T 9c962e4019054cffb50c200cb2fedf5f P be26fcebf49a43259cadc0f2b9c0325a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a4ce643e7f40d4b3ff4892217c5352" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 45357 } } peers { permanent_uuid: "be26fcebf49a43259cadc0f2b9c0325a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38903 } } peers { permanent_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36419 } }
I20260927 03:58:26.764120 18336 ts_tablet_manager.cc:1428] T 9c962e4019054cffb50c200cb2fedf5f P be26fcebf49a43259cadc0f2b9c0325a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 03:58:26.848901 18380 raft_consensus.cc:482] T 9c962e4019054cffb50c200cb2fedf5f P 1c62cb4ea2bf45a1830e7048550cf4db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:58:26.849049 18380 raft_consensus.cc:504] T 9c962e4019054cffb50c200cb2fedf5f P 1c62cb4ea2bf45a1830e7048550cf4db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a4ce643e7f40d4b3ff4892217c5352" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 45357 } } peers { permanent_uuid: "be26fcebf49a43259cadc0f2b9c0325a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38903 } } peers { permanent_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36419 } }
I20260927 03:58:26.849257 18380 leader_election.cc:288] T 9c962e4019054cffb50c200cb2fedf5f P 1c62cb4ea2bf45a1830e7048550cf4db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47a4ce643e7f40d4b3ff4892217c5352 (127.15.185.65:45357), be26fcebf49a43259cadc0f2b9c0325a (127.15.185.66:38903)
I20260927 03:58:26.849485 17982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c962e4019054cffb50c200cb2fedf5f" candidate_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47a4ce643e7f40d4b3ff4892217c5352" is_pre_election: true
I20260927 03:58:26.849488 18125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c962e4019054cffb50c200cb2fedf5f" candidate_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be26fcebf49a43259cadc0f2b9c0325a" is_pre_election: true
I20260927 03:58:26.849587 17982 raft_consensus.cc:2454] T 9c962e4019054cffb50c200cb2fedf5f P 47a4ce643e7f40d4b3ff4892217c5352 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c62cb4ea2bf45a1830e7048550cf4db in term 0.
I20260927 03:58:26.849589 18125 raft_consensus.cc:2454] T 9c962e4019054cffb50c200cb2fedf5f P be26fcebf49a43259cadc0f2b9c0325a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c62cb4ea2bf45a1830e7048550cf4db in term 0.
I20260927 03:58:26.849737 18199 leader_election.cc:302] T 9c962e4019054cffb50c200cb2fedf5f P 1c62cb4ea2bf45a1830e7048550cf4db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c62cb4ea2bf45a1830e7048550cf4db, be26fcebf49a43259cadc0f2b9c0325a; no voters:
I20260927 03:58:26.849843 18380 raft_consensus.cc:2781] T 9c962e4019054cffb50c200cb2fedf5f P 1c62cb4ea2bf45a1830e7048550cf4db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:58:26.849891 18380 raft_consensus.cc:482] T 9c962e4019054cffb50c200cb2fedf5f P 1c62cb4ea2bf45a1830e7048550cf4db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:58:26.849910 18380 raft_consensus.cc:3037] T 9c962e4019054cffb50c200cb2fedf5f P 1c62cb4ea2bf45a1830e7048550cf4db [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:26.850600 18380 raft_consensus.cc:504] T 9c962e4019054cffb50c200cb2fedf5f P 1c62cb4ea2bf45a1830e7048550cf4db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a4ce643e7f40d4b3ff4892217c5352" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 45357 } } peers { permanent_uuid: "be26fcebf49a43259cadc0f2b9c0325a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38903 } } peers { permanent_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36419 } }
I20260927 03:58:26.850735 18380 leader_election.cc:288] T 9c962e4019054cffb50c200cb2fedf5f P 1c62cb4ea2bf45a1830e7048550cf4db [CANDIDATE]: Term 1 election: Requested vote from peers 47a4ce643e7f40d4b3ff4892217c5352 (127.15.185.65:45357), be26fcebf49a43259cadc0f2b9c0325a (127.15.185.66:38903)
I20260927 03:58:26.850857 17982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c962e4019054cffb50c200cb2fedf5f" candidate_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47a4ce643e7f40d4b3ff4892217c5352"
I20260927 03:58:26.850929 17982 raft_consensus.cc:3037] T 9c962e4019054cffb50c200cb2fedf5f P 47a4ce643e7f40d4b3ff4892217c5352 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:26.850915 18125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c962e4019054cffb50c200cb2fedf5f" candidate_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be26fcebf49a43259cadc0f2b9c0325a"
I20260927 03:58:26.850984 18125 raft_consensus.cc:3037] T 9c962e4019054cffb50c200cb2fedf5f P be26fcebf49a43259cadc0f2b9c0325a [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:26.851727 18125 raft_consensus.cc:2454] T 9c962e4019054cffb50c200cb2fedf5f P be26fcebf49a43259cadc0f2b9c0325a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c62cb4ea2bf45a1830e7048550cf4db in term 1.
I20260927 03:58:26.851727 17982 raft_consensus.cc:2454] T 9c962e4019054cffb50c200cb2fedf5f P 47a4ce643e7f40d4b3ff4892217c5352 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c62cb4ea2bf45a1830e7048550cf4db in term 1.
I20260927 03:58:26.851907 18198 leader_election.cc:302] T 9c962e4019054cffb50c200cb2fedf5f P 1c62cb4ea2bf45a1830e7048550cf4db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1c62cb4ea2bf45a1830e7048550cf4db, 47a4ce643e7f40d4b3ff4892217c5352, be26fcebf49a43259cadc0f2b9c0325a; no voters:
I20260927 03:58:26.852000 18380 raft_consensus.cc:2781] T 9c962e4019054cffb50c200cb2fedf5f P 1c62cb4ea2bf45a1830e7048550cf4db [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:58:26.852077 18380 raft_consensus.cc:686] T 9c962e4019054cffb50c200cb2fedf5f P 1c62cb4ea2bf45a1830e7048550cf4db [term 1 LEADER]: Becoming Leader. State: Replica: 1c62cb4ea2bf45a1830e7048550cf4db, State: Running, Role: LEADER
I20260927 03:58:26.852128 18380 consensus_queue.cc:237] T 9c962e4019054cffb50c200cb2fedf5f P 1c62cb4ea2bf45a1830e7048550cf4db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a4ce643e7f40d4b3ff4892217c5352" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 45357 } } peers { permanent_uuid: "be26fcebf49a43259cadc0f2b9c0325a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38903 } } peers { permanent_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36419 } }
I20260927 03:58:26.852761 17827 catalog_manager.cc:5495] T 9c962e4019054cffb50c200cb2fedf5f P 1c62cb4ea2bf45a1830e7048550cf4db reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c62cb4ea2bf45a1830e7048550cf4db (127.15.185.67). New cstate: current_term: 1 leader_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a4ce643e7f40d4b3ff4892217c5352" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 45357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be26fcebf49a43259cadc0f2b9c0325a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36419 } health_report { overall_health: HEALTHY } } }
I20260927 03:58:26.878557 17982 raft_consensus.cc:1261] T 9c962e4019054cffb50c200cb2fedf5f P 47a4ce643e7f40d4b3ff4892217c5352 [term 1 FOLLOWER]: Refusing update from remote peer 1c62cb4ea2bf45a1830e7048550cf4db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:58:26.878610 18125 raft_consensus.cc:1261] T 9c962e4019054cffb50c200cb2fedf5f P be26fcebf49a43259cadc0f2b9c0325a [term 1 FOLLOWER]: Refusing update from remote peer 1c62cb4ea2bf45a1830e7048550cf4db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:58:26.878818 18375 consensus_queue.cc:1035] T 9c962e4019054cffb50c200cb2fedf5f P 1c62cb4ea2bf45a1830e7048550cf4db [LEADER]: Connected to new peer: Peer: permanent_uuid: "be26fcebf49a43259cadc0f2b9c0325a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:58:26.878931 18380 consensus_queue.cc:1035] T 9c962e4019054cffb50c200cb2fedf5f P 1c62cb4ea2bf45a1830e7048550cf4db [LEADER]: Connected to new peer: Peer: permanent_uuid: "47a4ce643e7f40d4b3ff4892217c5352" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 45357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 03:58:26.883230 18363 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 03:58:28.400616 17827 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36520:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:58:28.400743 17827 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36520:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:58:28.401588 17827 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 67a6d9bd1a4e47818b720084e11ccd8a: Sending DeleteTablet for 3 replicas of tablet 9c962e4019054cffb50c200cb2fedf5f
I20260927 03:58:28.401908 18105 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c962e4019054cffb50c200cb2fedf5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:58:28 UTC) from {username='kudu', principal='kudu/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:56438
I20260927 03:58:28.401932 18245 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c962e4019054cffb50c200cb2fedf5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:58:28 UTC) from {username='kudu', principal='kudu/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:41692
I20260927 03:58:28.401932 17965 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c962e4019054cffb50c200cb2fedf5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:58:28 UTC) from {username='kudu', principal='kudu/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:35292
I20260927 03:58:28.402117 18414 tablet_replica.cc:330] T 9c962e4019054cffb50c200cb2fedf5f P 1c62cb4ea2bf45a1830e7048550cf4db: stopping tablet replica
I20260927 03:58:28.402124 18412 tablet_replica.cc:330] T 9c962e4019054cffb50c200cb2fedf5f P be26fcebf49a43259cadc0f2b9c0325a: stopping tablet replica
I20260927 03:58:28.402172 18414 raft_consensus.cc:2229] T 9c962e4019054cffb50c200cb2fedf5f P 1c62cb4ea2bf45a1830e7048550cf4db [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:58:28.402189 18412 raft_consensus.cc:2229] T 9c962e4019054cffb50c200cb2fedf5f P be26fcebf49a43259cadc0f2b9c0325a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:58:28.402233 18414 raft_consensus.cc:2258] T 9c962e4019054cffb50c200cb2fedf5f P 1c62cb4ea2bf45a1830e7048550cf4db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:58:28.402243 18412 raft_consensus.cc:2258] T 9c962e4019054cffb50c200cb2fedf5f P be26fcebf49a43259cadc0f2b9c0325a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:58:28.402493 18412 ts_tablet_manager.cc:1905] T 9c962e4019054cffb50c200cb2fedf5f P be26fcebf49a43259cadc0f2b9c0325a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:58:28.402575 18414 ts_tablet_manager.cc:1905] T 9c962e4019054cffb50c200cb2fedf5f P 1c62cb4ea2bf45a1830e7048550cf4db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:58:28.403949 18413 tablet_replica.cc:330] T 9c962e4019054cffb50c200cb2fedf5f P 47a4ce643e7f40d4b3ff4892217c5352: stopping tablet replica
I20260927 03:58:28.404078 18413 raft_consensus.cc:2229] T 9c962e4019054cffb50c200cb2fedf5f P 47a4ce643e7f40d4b3ff4892217c5352 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:58:28.404222 18414 ts_tablet_manager.cc:1918] T 9c962e4019054cffb50c200cb2fedf5f P 1c62cb4ea2bf45a1830e7048550cf4db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 03:58:28.404314 18414 log.cc:1196] T 9c962e4019054cffb50c200cb2fedf5f P 1c62cb4ea2bf45a1830e7048550cf4db: Deleting WAL directory at /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/ts-2/wal/wals/9c962e4019054cffb50c200cb2fedf5f
I20260927 03:58:28.404641 18414 ts_tablet_manager.cc:1939] T 9c962e4019054cffb50c200cb2fedf5f P 1c62cb4ea2bf45a1830e7048550cf4db: Deleting consensus metadata
I20260927 03:58:28.404704 18412 ts_tablet_manager.cc:1918] T 9c962e4019054cffb50c200cb2fedf5f P be26fcebf49a43259cadc0f2b9c0325a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 03:58:28.404838 18412 log.cc:1196] T 9c962e4019054cffb50c200cb2fedf5f P be26fcebf49a43259cadc0f2b9c0325a: Deleting WAL directory at /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/ts-1/wal/wals/9c962e4019054cffb50c200cb2fedf5f
I20260927 03:58:28.405054 17812 catalog_manager.cc:4841] TS 1c62cb4ea2bf45a1830e7048550cf4db (127.15.185.67:36419): tablet 9c962e4019054cffb50c200cb2fedf5f (table test-table [id=2703ab1b1ef94908852b9234d0a44a82]) successfully deleted
I20260927 03:58:28.405153 18412 ts_tablet_manager.cc:1939] T 9c962e4019054cffb50c200cb2fedf5f P be26fcebf49a43259cadc0f2b9c0325a: Deleting consensus metadata
I20260927 03:58:28.405234 18413 raft_consensus.cc:2258] T 9c962e4019054cffb50c200cb2fedf5f P 47a4ce643e7f40d4b3ff4892217c5352 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:58:28.405540 17815 catalog_manager.cc:4841] TS be26fcebf49a43259cadc0f2b9c0325a (127.15.185.66:38903): tablet 9c962e4019054cffb50c200cb2fedf5f (table test-table [id=2703ab1b1ef94908852b9234d0a44a82]) successfully deleted
I20260927 03:58:28.405562 18413 ts_tablet_manager.cc:1905] T 9c962e4019054cffb50c200cb2fedf5f P 47a4ce643e7f40d4b3ff4892217c5352: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:58:28.406924 18413 ts_tablet_manager.cc:1918] T 9c962e4019054cffb50c200cb2fedf5f P 47a4ce643e7f40d4b3ff4892217c5352: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 03:58:28.406999 18413 log.cc:1196] T 9c962e4019054cffb50c200cb2fedf5f P 47a4ce643e7f40d4b3ff4892217c5352: Deleting WAL directory at /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/ts-0/wal/wals/9c962e4019054cffb50c200cb2fedf5f
I20260927 03:58:28.407222 18413 ts_tablet_manager.cc:1939] T 9c962e4019054cffb50c200cb2fedf5f P 47a4ce643e7f40d4b3ff4892217c5352: Deleting consensus metadata
I20260927 03:58:28.407557 17814 catalog_manager.cc:4841] TS 47a4ce643e7f40d4b3ff4892217c5352 (127.15.185.65:45357): tablet 9c962e4019054cffb50c200cb2fedf5f (table test-table [id=2703ab1b1ef94908852b9234d0a44a82]) successfully deleted
Sep 27 03:58:28 dist-test-slave-glgz krb5kdc[17783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481508, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 03:58:28 dist-test-slave-glgz krb5kdc[17783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481508, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 03:58:28.431509 18421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:28.420035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36532 (local address 127.15.185.126:41063)
0927 03:58:28.420202 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:28.420204 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:28.420283 (+ 79us) server_negotiation.cc:407] Connection header received
0927 03:58:28.420332 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:28.420333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:28.420345 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:28.420418 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:28.421037 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:28.421634 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:28.422355 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:28.422552 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:28.422609 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:28.429123 (+ 6514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:28.429125 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:28.429125 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:28.429133 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:28.430441 (+ 1308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:28.430754 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:28.430754 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:28.430755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:28.430808 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:28.430985 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:28.430986 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:28.430986 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:28.431143 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:28.431223 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:28.431327 (+ 104us) server_negotiation.cc:299] Negotiation successful
0927 03:58:28.431389 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":77,"threads_started":1}
I20260927 03:58:28.432986 17827 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36532:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:58:28.433130 17827 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:58:28.434530 18105 tablet_service.cc:1467] Processing CreateTablet for tablet 9db77b93b9c24d979f4fd7457b8ef94e (DEFAULT_TABLE table=test-table [id=abc3292b475849fd8871ac1fa3a6a4ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:58:28.434517 17965 tablet_service.cc:1467] Processing CreateTablet for tablet 9db77b93b9c24d979f4fd7457b8ef94e (DEFAULT_TABLE table=test-table [id=abc3292b475849fd8871ac1fa3a6a4ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:58:28.434617 18245 tablet_service.cc:1467] Processing CreateTablet for tablet 9db77b93b9c24d979f4fd7457b8ef94e (DEFAULT_TABLE table=test-table [id=abc3292b475849fd8871ac1fa3a6a4ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:58:28.434651 18105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9db77b93b9c24d979f4fd7457b8ef94e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:28.434656 17965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9db77b93b9c24d979f4fd7457b8ef94e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:28.434751 18245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9db77b93b9c24d979f4fd7457b8ef94e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:28.435992 18423 tablet_bootstrap.cc:492] T 9db77b93b9c24d979f4fd7457b8ef94e P 1c62cb4ea2bf45a1830e7048550cf4db: Bootstrap starting.
I20260927 03:58:28.436168 18424 tablet_bootstrap.cc:492] T 9db77b93b9c24d979f4fd7457b8ef94e P 47a4ce643e7f40d4b3ff4892217c5352: Bootstrap starting.
I20260927 03:58:28.436223 18425 tablet_bootstrap.cc:492] T 9db77b93b9c24d979f4fd7457b8ef94e P be26fcebf49a43259cadc0f2b9c0325a: Bootstrap starting.
I20260927 03:58:28.436329 18423 tablet_bootstrap.cc:654] T 9db77b93b9c24d979f4fd7457b8ef94e P 1c62cb4ea2bf45a1830e7048550cf4db: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:28.436548 18424 tablet_bootstrap.cc:654] T 9db77b93b9c24d979f4fd7457b8ef94e P 47a4ce643e7f40d4b3ff4892217c5352: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:28.436648 18425 tablet_bootstrap.cc:654] T 9db77b93b9c24d979f4fd7457b8ef94e P be26fcebf49a43259cadc0f2b9c0325a: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:28.436978 18423 tablet_bootstrap.cc:492] T 9db77b93b9c24d979f4fd7457b8ef94e P 1c62cb4ea2bf45a1830e7048550cf4db: No bootstrap required, opened a new log
I20260927 03:58:28.437040 18423 ts_tablet_manager.cc:1397] T 9db77b93b9c24d979f4fd7457b8ef94e P 1c62cb4ea2bf45a1830e7048550cf4db: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:58:28.437114 18424 tablet_bootstrap.cc:492] T 9db77b93b9c24d979f4fd7457b8ef94e P 47a4ce643e7f40d4b3ff4892217c5352: No bootstrap required, opened a new log
I20260927 03:58:28.437163 18424 ts_tablet_manager.cc:1397] T 9db77b93b9c24d979f4fd7457b8ef94e P 47a4ce643e7f40d4b3ff4892217c5352: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:58:28.437176 18425 tablet_bootstrap.cc:492] T 9db77b93b9c24d979f4fd7457b8ef94e P be26fcebf49a43259cadc0f2b9c0325a: No bootstrap required, opened a new log
I20260927 03:58:28.437161 18423 raft_consensus.cc:348] T 9db77b93b9c24d979f4fd7457b8ef94e P 1c62cb4ea2bf45a1830e7048550cf4db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a4ce643e7f40d4b3ff4892217c5352" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 45357 } } peers { permanent_uuid: "be26fcebf49a43259cadc0f2b9c0325a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38903 } } peers { permanent_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36419 } }
I20260927 03:58:28.437212 18425 ts_tablet_manager.cc:1397] T 9db77b93b9c24d979f4fd7457b8ef94e P be26fcebf49a43259cadc0f2b9c0325a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:58:28.437284 18423 raft_consensus.cc:374] T 9db77b93b9c24d979f4fd7457b8ef94e P 1c62cb4ea2bf45a1830e7048550cf4db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:28.437317 18423 raft_consensus.cc:729] T 9db77b93b9c24d979f4fd7457b8ef94e P 1c62cb4ea2bf45a1830e7048550cf4db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c62cb4ea2bf45a1830e7048550cf4db, State: Initialized, Role: FOLLOWER
I20260927 03:58:28.437295 18424 raft_consensus.cc:348] T 9db77b93b9c24d979f4fd7457b8ef94e P 47a4ce643e7f40d4b3ff4892217c5352 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a4ce643e7f40d4b3ff4892217c5352" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 45357 } } peers { permanent_uuid: "be26fcebf49a43259cadc0f2b9c0325a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38903 } } peers { permanent_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36419 } }
I20260927 03:58:28.437321 18425 raft_consensus.cc:348] T 9db77b93b9c24d979f4fd7457b8ef94e P be26fcebf49a43259cadc0f2b9c0325a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a4ce643e7f40d4b3ff4892217c5352" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 45357 } } peers { permanent_uuid: "be26fcebf49a43259cadc0f2b9c0325a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38903 } } peers { permanent_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36419 } }
I20260927 03:58:28.437363 18424 raft_consensus.cc:374] T 9db77b93b9c24d979f4fd7457b8ef94e P 47a4ce643e7f40d4b3ff4892217c5352 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:28.437391 18424 raft_consensus.cc:729] T 9db77b93b9c24d979f4fd7457b8ef94e P 47a4ce643e7f40d4b3ff4892217c5352 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47a4ce643e7f40d4b3ff4892217c5352, State: Initialized, Role: FOLLOWER
I20260927 03:58:28.437394 18425 raft_consensus.cc:374] T 9db77b93b9c24d979f4fd7457b8ef94e P be26fcebf49a43259cadc0f2b9c0325a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:28.437427 18425 raft_consensus.cc:729] T 9db77b93b9c24d979f4fd7457b8ef94e P be26fcebf49a43259cadc0f2b9c0325a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be26fcebf49a43259cadc0f2b9c0325a, State: Initialized, Role: FOLLOWER
I20260927 03:58:28.437451 18424 consensus_queue.cc:260] T 9db77b93b9c24d979f4fd7457b8ef94e P 47a4ce643e7f40d4b3ff4892217c5352 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a4ce643e7f40d4b3ff4892217c5352" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 45357 } } peers { permanent_uuid: "be26fcebf49a43259cadc0f2b9c0325a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38903 } } peers { permanent_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36419 } }
I20260927 03:58:28.437470 18423 consensus_queue.cc:260] T 9db77b93b9c24d979f4fd7457b8ef94e P 1c62cb4ea2bf45a1830e7048550cf4db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a4ce643e7f40d4b3ff4892217c5352" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 45357 } } peers { permanent_uuid: "be26fcebf49a43259cadc0f2b9c0325a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38903 } } peers { permanent_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36419 } }
I20260927 03:58:28.437510 18425 consensus_queue.cc:260] T 9db77b93b9c24d979f4fd7457b8ef94e P be26fcebf49a43259cadc0f2b9c0325a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a4ce643e7f40d4b3ff4892217c5352" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 45357 } } peers { permanent_uuid: "be26fcebf49a43259cadc0f2b9c0325a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38903 } } peers { permanent_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36419 } }
I20260927 03:58:28.437614 18425 ts_tablet_manager.cc:1428] T 9db77b93b9c24d979f4fd7457b8ef94e P be26fcebf49a43259cadc0f2b9c0325a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:28.437667 18423 ts_tablet_manager.cc:1428] T 9db77b93b9c24d979f4fd7457b8ef94e P 1c62cb4ea2bf45a1830e7048550cf4db: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:58:28.437671 18424 ts_tablet_manager.cc:1428] T 9db77b93b9c24d979f4fd7457b8ef94e P 47a4ce643e7f40d4b3ff4892217c5352: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 03:58:28.745683 18429 raft_consensus.cc:482] T 9db77b93b9c24d979f4fd7457b8ef94e P 47a4ce643e7f40d4b3ff4892217c5352 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:58:28.745783 18429 raft_consensus.cc:504] T 9db77b93b9c24d979f4fd7457b8ef94e P 47a4ce643e7f40d4b3ff4892217c5352 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a4ce643e7f40d4b3ff4892217c5352" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 45357 } } peers { permanent_uuid: "be26fcebf49a43259cadc0f2b9c0325a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38903 } } peers { permanent_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36419 } }
I20260927 03:58:28.746182 18429 leader_election.cc:288] T 9db77b93b9c24d979f4fd7457b8ef94e P 47a4ce643e7f40d4b3ff4892217c5352 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be26fcebf49a43259cadc0f2b9c0325a (127.15.185.66:38903), 1c62cb4ea2bf45a1830e7048550cf4db (127.15.185.67:36419)
I20260927 03:58:28.749285 18433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:28.746508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:36497 (local address 127.15.185.67:36419)
0927 03:58:28.746644 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:28.746646 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:28.746657 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:58:28.746697 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:28.746698 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:28.746710 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:28.746793 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:58:28.746955 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:28.747499 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:28.748548 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:28.748922 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:28.749007 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:28.749016 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:28.749170 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 03:58:28.749211 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":73,"threads_started":1}
I20260927 03:58:28.749294 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:28.746404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:39953 (local address 127.15.185.66:38903)
0927 03:58:28.746552 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:28.746554 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:28.746566 (+ 12us) server_negotiation.cc:407] Connection header received
0927 03:58:28.746657 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:28.746658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:28.746670 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:28.746770 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:58:28.746945 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:28.747519 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:28.748548 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:28.748937 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:28.749023 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:28.749030 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:28.749182 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 03:58:28.749221 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":87,"threads_started":1}
I20260927 03:58:28.749321 18430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:28.746341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.66:38903 (local address 127.15.185.65:39953)
0927 03:58:28.746501 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:28.746513 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:28.746577 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:28.746778 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:28.746779 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:28.746781 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:58:28.746856 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:28.746858 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:28.747601 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:28.747602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:28.748451 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:28.748453 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:28.749022 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:28.749023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:28.749103 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:28.749106 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:58:28.749176 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 03:58:28.749218 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":27,"thread_start_us":106,"threads_started":1}
I20260927 03:58:28.749496 18432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:28.746385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.67:36419 (local address 127.15.185.65:36497)
0927 03:58:28.746557 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:28.746567 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:28.746636 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:28.746795 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:28.746796 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:28.746798 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:58:28.746871 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:28.746872 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:28.747591 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:28.747592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:28.748450 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:28.748452 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:28.749004 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:28.749006 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:28.749074 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:28.749078 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:58:28.749151 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 03:58:28.749194 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":45,"thread_start_us":106,"threads_started":1}
I20260927 03:58:28.749745 18265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9db77b93b9c24d979f4fd7457b8ef94e" candidate_uuid: "47a4ce643e7f40d4b3ff4892217c5352" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" is_pre_election: true
I20260927 03:58:28.749873 18265 raft_consensus.cc:2454] T 9db77b93b9c24d979f4fd7457b8ef94e P 1c62cb4ea2bf45a1830e7048550cf4db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47a4ce643e7f40d4b3ff4892217c5352 in term 0.
I20260927 03:58:28.749833 18125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9db77b93b9c24d979f4fd7457b8ef94e" candidate_uuid: "47a4ce643e7f40d4b3ff4892217c5352" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be26fcebf49a43259cadc0f2b9c0325a" is_pre_election: true
I20260927 03:58:28.749920 18125 raft_consensus.cc:2454] T 9db77b93b9c24d979f4fd7457b8ef94e P be26fcebf49a43259cadc0f2b9c0325a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47a4ce643e7f40d4b3ff4892217c5352 in term 0.
I20260927 03:58:28.750077 17916 leader_election.cc:302] T 9db77b93b9c24d979f4fd7457b8ef94e P 47a4ce643e7f40d4b3ff4892217c5352 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c62cb4ea2bf45a1830e7048550cf4db, 47a4ce643e7f40d4b3ff4892217c5352; no voters:
I20260927 03:58:28.750222 18429 raft_consensus.cc:2781] T 9db77b93b9c24d979f4fd7457b8ef94e P 47a4ce643e7f40d4b3ff4892217c5352 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:58:28.750262 18429 raft_consensus.cc:482] T 9db77b93b9c24d979f4fd7457b8ef94e P 47a4ce643e7f40d4b3ff4892217c5352 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:58:28.750288 18429 raft_consensus.cc:3037] T 9db77b93b9c24d979f4fd7457b8ef94e P 47a4ce643e7f40d4b3ff4892217c5352 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:28.750980 18429 raft_consensus.cc:504] T 9db77b93b9c24d979f4fd7457b8ef94e P 47a4ce643e7f40d4b3ff4892217c5352 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a4ce643e7f40d4b3ff4892217c5352" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 45357 } } peers { permanent_uuid: "be26fcebf49a43259cadc0f2b9c0325a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38903 } } peers { permanent_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36419 } }
I20260927 03:58:28.751163 18429 leader_election.cc:288] T 9db77b93b9c24d979f4fd7457b8ef94e P 47a4ce643e7f40d4b3ff4892217c5352 [CANDIDATE]: Term 1 election: Requested vote from peers be26fcebf49a43259cadc0f2b9c0325a (127.15.185.66:38903), 1c62cb4ea2bf45a1830e7048550cf4db (127.15.185.67:36419)
I20260927 03:58:28.751308 18125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9db77b93b9c24d979f4fd7457b8ef94e" candidate_uuid: "47a4ce643e7f40d4b3ff4892217c5352" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be26fcebf49a43259cadc0f2b9c0325a"
I20260927 03:58:28.751396 18125 raft_consensus.cc:3037] T 9db77b93b9c24d979f4fd7457b8ef94e P be26fcebf49a43259cadc0f2b9c0325a [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:28.751471 18265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9db77b93b9c24d979f4fd7457b8ef94e" candidate_uuid: "47a4ce643e7f40d4b3ff4892217c5352" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c62cb4ea2bf45a1830e7048550cf4db"
I20260927 03:58:28.751550 18265 raft_consensus.cc:3037] T 9db77b93b9c24d979f4fd7457b8ef94e P 1c62cb4ea2bf45a1830e7048550cf4db [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:28.752066 18125 raft_consensus.cc:2454] T 9db77b93b9c24d979f4fd7457b8ef94e P be26fcebf49a43259cadc0f2b9c0325a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47a4ce643e7f40d4b3ff4892217c5352 in term 1.
I20260927 03:58:28.752142 18265 raft_consensus.cc:2454] T 9db77b93b9c24d979f4fd7457b8ef94e P 1c62cb4ea2bf45a1830e7048550cf4db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47a4ce643e7f40d4b3ff4892217c5352 in term 1.
I20260927 03:58:28.752270 17919 leader_election.cc:302] T 9db77b93b9c24d979f4fd7457b8ef94e P 47a4ce643e7f40d4b3ff4892217c5352 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47a4ce643e7f40d4b3ff4892217c5352, be26fcebf49a43259cadc0f2b9c0325a; no voters:
I20260927 03:58:28.752367 18429 raft_consensus.cc:2781] T 9db77b93b9c24d979f4fd7457b8ef94e P 47a4ce643e7f40d4b3ff4892217c5352 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:58:28.752517 18429 raft_consensus.cc:686] T 9db77b93b9c24d979f4fd7457b8ef94e P 47a4ce643e7f40d4b3ff4892217c5352 [term 1 LEADER]: Becoming Leader. State: Replica: 47a4ce643e7f40d4b3ff4892217c5352, State: Running, Role: LEADER
I20260927 03:58:28.752606 18429 consensus_queue.cc:237] T 9db77b93b9c24d979f4fd7457b8ef94e P 47a4ce643e7f40d4b3ff4892217c5352 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a4ce643e7f40d4b3ff4892217c5352" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 45357 } } peers { permanent_uuid: "be26fcebf49a43259cadc0f2b9c0325a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38903 } } peers { permanent_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36419 } }
I20260927 03:58:28.753352 17828 catalog_manager.cc:5495] T 9db77b93b9c24d979f4fd7457b8ef94e P 47a4ce643e7f40d4b3ff4892217c5352 reported cstate change: term changed from 0 to 1, leader changed from <none> to 47a4ce643e7f40d4b3ff4892217c5352 (127.15.185.65). New cstate: current_term: 1 leader_uuid: "47a4ce643e7f40d4b3ff4892217c5352" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a4ce643e7f40d4b3ff4892217c5352" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 45357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be26fcebf49a43259cadc0f2b9c0325a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36419 } health_report { overall_health: UNKNOWN } } }
I20260927 03:58:28.794822 18437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:28.792528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35302 (local address 127.15.185.65:45357)
0927 03:58:28.792689 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:28.792691 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:28.792699 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:58:28.792747 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:28.792747 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:28.792767 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:28.792846 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:58:28.792931 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:28.793416 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:28.794142 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:28.794342 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:28.794393 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:28.794535 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:58:28.794640 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:58:28.794693 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:28.794732 (+ 39us) server_negotiation.cc:299] Negotiation successful
0927 03:58:28.794748 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":106,"threads_started":1}
I20260927 03:58:28.799827 18440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:28.797424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.66:38903 (local address 127.15.185.65:35219)
0927 03:58:28.797590 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:28.797599 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:28.797650 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:28.797752 (+ 102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:28.797752 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:28.797754 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 03:58:28.797808 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:28.797809 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:28.798405 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:28.798406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:28.798998 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:28.799000 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:28.799355 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:28.799356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:28.799405 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:28.799418 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 03:58:28.799679 (+ 261us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 03:58:28.799681 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 03:58:28.799723 (+ 42us) client_negotiation.cc:240] Negotiation successful
0927 03:58:28.799759 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":82,"threads_started":1}
I20260927 03:58:28.799824 18431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:28.797526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:35219 (local address 127.15.185.66:38903)
0927 03:58:28.797580 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:28.797583 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:28.797642 (+ 59us) server_negotiation.cc:407] Connection header received
0927 03:58:28.797691 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:28.797692 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:28.797700 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:28.797752 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:58:28.797843 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:28.798335 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:28.799080 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:28.799286 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:28.799342 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:28.799540 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:58:28.799574 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:58:28.799626 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:28.799747 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 03:58:28.799766 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 03:58:28.802071 18265 raft_consensus.cc:1261] T 9db77b93b9c24d979f4fd7457b8ef94e P 1c62cb4ea2bf45a1830e7048550cf4db [term 1 FOLLOWER]: Refusing update from remote peer 47a4ce643e7f40d4b3ff4892217c5352: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:58:28.802080 18125 raft_consensus.cc:1261] T 9db77b93b9c24d979f4fd7457b8ef94e P be26fcebf49a43259cadc0f2b9c0325a [term 1 FOLLOWER]: Refusing update from remote peer 47a4ce643e7f40d4b3ff4892217c5352: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:58:28.802340 18429 consensus_queue.cc:1035] T 9db77b93b9c24d979f4fd7457b8ef94e P 47a4ce643e7f40d4b3ff4892217c5352 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:58:28.802516 18434 consensus_queue.cc:1035] T 9db77b93b9c24d979f4fd7457b8ef94e P 47a4ce643e7f40d4b3ff4892217c5352 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be26fcebf49a43259cadc0f2b9c0325a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:58:28.810212 18445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:28.807819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.65:45357 (local address 127.15.185.66:48359)
0927 03:58:28.807947 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:28.807959 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:28.808018 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:28.808173 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:28.808174 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:28.808176 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 03:58:28.808228 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:28.808230 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:28.808881 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:28.808882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:28.809425 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:28.809427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:28.809803 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:28.809804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:28.809862 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:28.809867 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 03:58:28.810037 (+ 170us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 03:58:28.810039 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 03:58:28.810082 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 03:58:28.810125 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":83,"threads_started":1}
I20260927 03:58:28.810213 18437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:28.807886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:48359 (local address 127.15.185.65:45357)
0927 03:58:28.807925 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:28.807927 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:28.808011 (+ 84us) server_negotiation.cc:407] Connection header received
0927 03:58:28.808080 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:28.808080 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:28.808088 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:28.808168 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:58:28.808296 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:28.808810 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:28.809518 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:28.809701 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:28.809749 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:28.809940 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:58:28.809972 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:58:28.810025 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:28.810096 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 03:58:28.810119 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 03:58:28.816824 17828 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:58:28.816937 17828 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:58:28.817579 17828 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 67a6d9bd1a4e47818b720084e11ccd8a: Sending DeleteTablet for 3 replicas of tablet 9db77b93b9c24d979f4fd7457b8ef94e
I20260927 03:58:28.817858 17965 tablet_service.cc:1514] Processing DeleteTablet for tablet 9db77b93b9c24d979f4fd7457b8ef94e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:58:28 UTC) from {username='kudu', principal='kudu/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:35292
I20260927 03:58:28.817855 18245 tablet_service.cc:1514] Processing DeleteTablet for tablet 9db77b93b9c24d979f4fd7457b8ef94e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:58:28 UTC) from {username='kudu', principal='kudu/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:41692
I20260927 03:58:28.817968 18413 tablet_replica.cc:330] T 9db77b93b9c24d979f4fd7457b8ef94e P 47a4ce643e7f40d4b3ff4892217c5352: stopping tablet replica
I20260927 03:58:28.818023 18413 raft_consensus.cc:2229] T 9db77b93b9c24d979f4fd7457b8ef94e P 47a4ce643e7f40d4b3ff4892217c5352 [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:58:28.818095 18413 raft_consensus.cc:2258] T 9db77b93b9c24d979f4fd7457b8ef94e P 47a4ce643e7f40d4b3ff4892217c5352 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:58:28.818038 17828 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36532:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 03:58:28.818089 18105 tablet_service.cc:1514] Processing DeleteTablet for tablet 9db77b93b9c24d979f4fd7457b8ef94e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:58:28 UTC) from {username='kudu', principal='kudu/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:56438
W20260927 03:58:28.818156 17828 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:58:28.818172 18414 tablet_replica.cc:330] T 9db77b93b9c24d979f4fd7457b8ef94e P 1c62cb4ea2bf45a1830e7048550cf4db: stopping tablet replica
I20260927 03:58:28.818183 18412 tablet_replica.cc:330] T 9db77b93b9c24d979f4fd7457b8ef94e P be26fcebf49a43259cadc0f2b9c0325a: stopping tablet replica
I20260927 03:58:28.818226 18414 raft_consensus.cc:2229] T 9db77b93b9c24d979f4fd7457b8ef94e P 1c62cb4ea2bf45a1830e7048550cf4db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:58:28.818235 18412 raft_consensus.cc:2229] T 9db77b93b9c24d979f4fd7457b8ef94e P be26fcebf49a43259cadc0f2b9c0325a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:58:28.818259 18414 raft_consensus.cc:2258] T 9db77b93b9c24d979f4fd7457b8ef94e P 1c62cb4ea2bf45a1830e7048550cf4db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:58:28.818274 18412 raft_consensus.cc:2258] T 9db77b93b9c24d979f4fd7457b8ef94e P be26fcebf49a43259cadc0f2b9c0325a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:58:28.818632 18414 ts_tablet_manager.cc:1905] T 9db77b93b9c24d979f4fd7457b8ef94e P 1c62cb4ea2bf45a1830e7048550cf4db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:58:28.818723 18413 ts_tablet_manager.cc:1905] T 9db77b93b9c24d979f4fd7457b8ef94e P 47a4ce643e7f40d4b3ff4892217c5352: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:58:28.818821 18412 ts_tablet_manager.cc:1905] T 9db77b93b9c24d979f4fd7457b8ef94e P be26fcebf49a43259cadc0f2b9c0325a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:58:28.820585 18413 ts_tablet_manager.cc:1918] T 9db77b93b9c24d979f4fd7457b8ef94e P 47a4ce643e7f40d4b3ff4892217c5352: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:58:28.820642 18413 log.cc:1196] T 9db77b93b9c24d979f4fd7457b8ef94e P 47a4ce643e7f40d4b3ff4892217c5352: Deleting WAL directory at /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/ts-0/wal/wals/9db77b93b9c24d979f4fd7457b8ef94e
I20260927 03:58:28.820860 18413 ts_tablet_manager.cc:1939] T 9db77b93b9c24d979f4fd7457b8ef94e P 47a4ce643e7f40d4b3ff4892217c5352: Deleting consensus metadata
I20260927 03:58:28.820919 18412 ts_tablet_manager.cc:1918] T 9db77b93b9c24d979f4fd7457b8ef94e P be26fcebf49a43259cadc0f2b9c0325a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:58:28.820971 18412 log.cc:1196] T 9db77b93b9c24d979f4fd7457b8ef94e P be26fcebf49a43259cadc0f2b9c0325a: Deleting WAL directory at /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/ts-1/wal/wals/9db77b93b9c24d979f4fd7457b8ef94e
I20260927 03:58:28.821139 17814 catalog_manager.cc:4841] TS 47a4ce643e7f40d4b3ff4892217c5352 (127.15.185.65:45357): tablet 9db77b93b9c24d979f4fd7457b8ef94e (table test-table [id=abc3292b475849fd8871ac1fa3a6a4ec]) successfully deleted
I20260927 03:58:28.821213 18412 ts_tablet_manager.cc:1939] T 9db77b93b9c24d979f4fd7457b8ef94e P be26fcebf49a43259cadc0f2b9c0325a: Deleting consensus metadata
I20260927 03:58:28.821272 17965 tablet_service.cc:1467] Processing CreateTablet for tablet 55c979b5d6e64481959e2e65e319fc9a (DEFAULT_TABLE table=test-table [id=774f977571b9484c8f1c21f4cda5962c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:58:28.821372 17965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55c979b5d6e64481959e2e65e319fc9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:28.822209 18424 tablet_bootstrap.cc:492] T 55c979b5d6e64481959e2e65e319fc9a P 47a4ce643e7f40d4b3ff4892217c5352: Bootstrap starting.
I20260927 03:58:28.822499 18245 tablet_service.cc:1467] Processing CreateTablet for tablet 55c979b5d6e64481959e2e65e319fc9a (DEFAULT_TABLE table=test-table [id=774f977571b9484c8f1c21f4cda5962c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:58:28.822618 18424 tablet_bootstrap.cc:654] T 55c979b5d6e64481959e2e65e319fc9a P 47a4ce643e7f40d4b3ff4892217c5352: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:28.822691 17815 catalog_manager.cc:4841] TS be26fcebf49a43259cadc0f2b9c0325a (127.15.185.66:38903): tablet 9db77b93b9c24d979f4fd7457b8ef94e (table test-table [id=abc3292b475849fd8871ac1fa3a6a4ec]) successfully deleted
I20260927 03:58:28.822824 18414 ts_tablet_manager.cc:1918] T 9db77b93b9c24d979f4fd7457b8ef94e P 1c62cb4ea2bf45a1830e7048550cf4db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:58:28.822870 18414 log.cc:1196] T 9db77b93b9c24d979f4fd7457b8ef94e P 1c62cb4ea2bf45a1830e7048550cf4db: Deleting WAL directory at /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/ts-2/wal/wals/9db77b93b9c24d979f4fd7457b8ef94e
I20260927 03:58:28.823014 18105 tablet_service.cc:1467] Processing CreateTablet for tablet 55c979b5d6e64481959e2e65e319fc9a (DEFAULT_TABLE table=test-table [id=774f977571b9484c8f1c21f4cda5962c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:58:28.823119 18414 ts_tablet_manager.cc:1939] T 9db77b93b9c24d979f4fd7457b8ef94e P 1c62cb4ea2bf45a1830e7048550cf4db: Deleting consensus metadata
I20260927 03:58:28.823140 18105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55c979b5d6e64481959e2e65e319fc9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:28.823251 18245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55c979b5d6e64481959e2e65e319fc9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:28.824389 18423 tablet_bootstrap.cc:492] T 55c979b5d6e64481959e2e65e319fc9a P 1c62cb4ea2bf45a1830e7048550cf4db: Bootstrap starting.
I20260927 03:58:28.824376 18425 tablet_bootstrap.cc:492] T 55c979b5d6e64481959e2e65e319fc9a P be26fcebf49a43259cadc0f2b9c0325a: Bootstrap starting.
I20260927 03:58:28.824702 18425 tablet_bootstrap.cc:654] T 55c979b5d6e64481959e2e65e319fc9a P be26fcebf49a43259cadc0f2b9c0325a: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:28.824740 18423 tablet_bootstrap.cc:654] T 55c979b5d6e64481959e2e65e319fc9a P 1c62cb4ea2bf45a1830e7048550cf4db: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:28.824862 18424 tablet_bootstrap.cc:492] T 55c979b5d6e64481959e2e65e319fc9a P 47a4ce643e7f40d4b3ff4892217c5352: No bootstrap required, opened a new log
I20260927 03:58:28.824908 18424 ts_tablet_manager.cc:1397] T 55c979b5d6e64481959e2e65e319fc9a P 47a4ce643e7f40d4b3ff4892217c5352: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 03:58:28.825093 18424 raft_consensus.cc:348] T 55c979b5d6e64481959e2e65e319fc9a P 47a4ce643e7f40d4b3ff4892217c5352 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a4ce643e7f40d4b3ff4892217c5352" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 45357 } } peers { permanent_uuid: "be26fcebf49a43259cadc0f2b9c0325a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38903 } } peers { permanent_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36419 } }
I20260927 03:58:28.825181 18424 raft_consensus.cc:374] T 55c979b5d6e64481959e2e65e319fc9a P 47a4ce643e7f40d4b3ff4892217c5352 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:28.825203 18424 raft_consensus.cc:729] T 55c979b5d6e64481959e2e65e319fc9a P 47a4ce643e7f40d4b3ff4892217c5352 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47a4ce643e7f40d4b3ff4892217c5352, State: Initialized, Role: FOLLOWER
I20260927 03:58:28.825243 18424 consensus_queue.cc:260] T 55c979b5d6e64481959e2e65e319fc9a P 47a4ce643e7f40d4b3ff4892217c5352 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a4ce643e7f40d4b3ff4892217c5352" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 45357 } } peers { permanent_uuid: "be26fcebf49a43259cadc0f2b9c0325a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38903 } } peers { permanent_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36419 } }
I20260927 03:58:28.825374 18424 ts_tablet_manager.cc:1428] T 55c979b5d6e64481959e2e65e319fc9a P 47a4ce643e7f40d4b3ff4892217c5352: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:28.825380 17812 catalog_manager.cc:4841] TS 1c62cb4ea2bf45a1830e7048550cf4db (127.15.185.67:36419): tablet 9db77b93b9c24d979f4fd7457b8ef94e (table test-table [id=abc3292b475849fd8871ac1fa3a6a4ec]) successfully deleted
I20260927 03:58:28.825886 18423 tablet_bootstrap.cc:492] T 55c979b5d6e64481959e2e65e319fc9a P 1c62cb4ea2bf45a1830e7048550cf4db: No bootstrap required, opened a new log
I20260927 03:58:28.825937 18423 ts_tablet_manager.cc:1397] T 55c979b5d6e64481959e2e65e319fc9a P 1c62cb4ea2bf45a1830e7048550cf4db: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:58:28.826043 18425 tablet_bootstrap.cc:492] T 55c979b5d6e64481959e2e65e319fc9a P be26fcebf49a43259cadc0f2b9c0325a: No bootstrap required, opened a new log
I20260927 03:58:28.826086 18425 ts_tablet_manager.cc:1397] T 55c979b5d6e64481959e2e65e319fc9a P be26fcebf49a43259cadc0f2b9c0325a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:58:28.826053 18423 raft_consensus.cc:348] T 55c979b5d6e64481959e2e65e319fc9a P 1c62cb4ea2bf45a1830e7048550cf4db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a4ce643e7f40d4b3ff4892217c5352" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 45357 } } peers { permanent_uuid: "be26fcebf49a43259cadc0f2b9c0325a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38903 } } peers { permanent_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36419 } }
I20260927 03:58:28.826128 18423 raft_consensus.cc:374] T 55c979b5d6e64481959e2e65e319fc9a P 1c62cb4ea2bf45a1830e7048550cf4db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:28.826152 18423 raft_consensus.cc:729] T 55c979b5d6e64481959e2e65e319fc9a P 1c62cb4ea2bf45a1830e7048550cf4db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c62cb4ea2bf45a1830e7048550cf4db, State: Initialized, Role: FOLLOWER
I20260927 03:58:28.826195 18423 consensus_queue.cc:260] T 55c979b5d6e64481959e2e65e319fc9a P 1c62cb4ea2bf45a1830e7048550cf4db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a4ce643e7f40d4b3ff4892217c5352" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 45357 } } peers { permanent_uuid: "be26fcebf49a43259cadc0f2b9c0325a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38903 } } peers { permanent_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36419 } }
I20260927 03:58:28.826206 18425 raft_consensus.cc:348] T 55c979b5d6e64481959e2e65e319fc9a P be26fcebf49a43259cadc0f2b9c0325a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a4ce643e7f40d4b3ff4892217c5352" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 45357 } } peers { permanent_uuid: "be26fcebf49a43259cadc0f2b9c0325a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38903 } } peers { permanent_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36419 } }
I20260927 03:58:28.826277 18425 raft_consensus.cc:374] T 55c979b5d6e64481959e2e65e319fc9a P be26fcebf49a43259cadc0f2b9c0325a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:28.826297 18425 raft_consensus.cc:729] T 55c979b5d6e64481959e2e65e319fc9a P be26fcebf49a43259cadc0f2b9c0325a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be26fcebf49a43259cadc0f2b9c0325a, State: Initialized, Role: FOLLOWER
I20260927 03:58:28.826293 18423 ts_tablet_manager.cc:1428] T 55c979b5d6e64481959e2e65e319fc9a P 1c62cb4ea2bf45a1830e7048550cf4db: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:28.826337 18425 consensus_queue.cc:260] T 55c979b5d6e64481959e2e65e319fc9a P be26fcebf49a43259cadc0f2b9c0325a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a4ce643e7f40d4b3ff4892217c5352" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 45357 } } peers { permanent_uuid: "be26fcebf49a43259cadc0f2b9c0325a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38903 } } peers { permanent_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36419 } }
I20260927 03:58:28.826428 18425 ts_tablet_manager.cc:1428] T 55c979b5d6e64481959e2e65e319fc9a P be26fcebf49a43259cadc0f2b9c0325a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:28.842576 18403 raft_consensus.cc:482] T 55c979b5d6e64481959e2e65e319fc9a P 1c62cb4ea2bf45a1830e7048550cf4db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:58:28.842665 18403 raft_consensus.cc:504] T 55c979b5d6e64481959e2e65e319fc9a P 1c62cb4ea2bf45a1830e7048550cf4db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a4ce643e7f40d4b3ff4892217c5352" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 45357 } } peers { permanent_uuid: "be26fcebf49a43259cadc0f2b9c0325a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38903 } } peers { permanent_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36419 } }
I20260927 03:58:28.842826 18403 leader_election.cc:288] T 55c979b5d6e64481959e2e65e319fc9a P 1c62cb4ea2bf45a1830e7048550cf4db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47a4ce643e7f40d4b3ff4892217c5352 (127.15.185.65:45357), be26fcebf49a43259cadc0f2b9c0325a (127.15.185.66:38903)
I20260927 03:58:28.843010 18125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55c979b5d6e64481959e2e65e319fc9a" candidate_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be26fcebf49a43259cadc0f2b9c0325a" is_pre_election: true
I20260927 03:58:28.843132 18125 raft_consensus.cc:2454] T 55c979b5d6e64481959e2e65e319fc9a P be26fcebf49a43259cadc0f2b9c0325a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c62cb4ea2bf45a1830e7048550cf4db in term 0.
I20260927 03:58:28.843035 17982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55c979b5d6e64481959e2e65e319fc9a" candidate_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47a4ce643e7f40d4b3ff4892217c5352" is_pre_election: true
I20260927 03:58:28.843196 17982 raft_consensus.cc:2454] T 55c979b5d6e64481959e2e65e319fc9a P 47a4ce643e7f40d4b3ff4892217c5352 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c62cb4ea2bf45a1830e7048550cf4db in term 0.
I20260927 03:58:28.843269 18199 leader_election.cc:302] T 55c979b5d6e64481959e2e65e319fc9a P 1c62cb4ea2bf45a1830e7048550cf4db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c62cb4ea2bf45a1830e7048550cf4db, be26fcebf49a43259cadc0f2b9c0325a; no voters:
I20260927 03:58:28.843370 18403 raft_consensus.cc:2781] T 55c979b5d6e64481959e2e65e319fc9a P 1c62cb4ea2bf45a1830e7048550cf4db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:58:28.843410 18403 raft_consensus.cc:482] T 55c979b5d6e64481959e2e65e319fc9a P 1c62cb4ea2bf45a1830e7048550cf4db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:58:28.843430 18403 raft_consensus.cc:3037] T 55c979b5d6e64481959e2e65e319fc9a P 1c62cb4ea2bf45a1830e7048550cf4db [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:28.844314 18403 raft_consensus.cc:504] T 55c979b5d6e64481959e2e65e319fc9a P 1c62cb4ea2bf45a1830e7048550cf4db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a4ce643e7f40d4b3ff4892217c5352" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 45357 } } peers { permanent_uuid: "be26fcebf49a43259cadc0f2b9c0325a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38903 } } peers { permanent_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36419 } }
I20260927 03:58:28.844451 18403 leader_election.cc:288] T 55c979b5d6e64481959e2e65e319fc9a P 1c62cb4ea2bf45a1830e7048550cf4db [CANDIDATE]: Term 1 election: Requested vote from peers 47a4ce643e7f40d4b3ff4892217c5352 (127.15.185.65:45357), be26fcebf49a43259cadc0f2b9c0325a (127.15.185.66:38903)
I20260927 03:58:28.844594 18125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55c979b5d6e64481959e2e65e319fc9a" candidate_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be26fcebf49a43259cadc0f2b9c0325a"
I20260927 03:58:28.844594 17982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55c979b5d6e64481959e2e65e319fc9a" candidate_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47a4ce643e7f40d4b3ff4892217c5352"
I20260927 03:58:28.844666 17982 raft_consensus.cc:3037] T 55c979b5d6e64481959e2e65e319fc9a P 47a4ce643e7f40d4b3ff4892217c5352 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:28.844666 18125 raft_consensus.cc:3037] T 55c979b5d6e64481959e2e65e319fc9a P be26fcebf49a43259cadc0f2b9c0325a [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:28.845418 18125 raft_consensus.cc:2454] T 55c979b5d6e64481959e2e65e319fc9a P be26fcebf49a43259cadc0f2b9c0325a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c62cb4ea2bf45a1830e7048550cf4db in term 1.
I20260927 03:58:28.845418 17982 raft_consensus.cc:2454] T 55c979b5d6e64481959e2e65e319fc9a P 47a4ce643e7f40d4b3ff4892217c5352 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c62cb4ea2bf45a1830e7048550cf4db in term 1.
I20260927 03:58:28.845577 18198 leader_election.cc:302] T 55c979b5d6e64481959e2e65e319fc9a P 1c62cb4ea2bf45a1830e7048550cf4db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c62cb4ea2bf45a1830e7048550cf4db, 47a4ce643e7f40d4b3ff4892217c5352; no voters:
I20260927 03:58:28.845676 18403 raft_consensus.cc:2781] T 55c979b5d6e64481959e2e65e319fc9a P 1c62cb4ea2bf45a1830e7048550cf4db [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:58:28.845739 18403 raft_consensus.cc:686] T 55c979b5d6e64481959e2e65e319fc9a P 1c62cb4ea2bf45a1830e7048550cf4db [term 1 LEADER]: Becoming Leader. State: Replica: 1c62cb4ea2bf45a1830e7048550cf4db, State: Running, Role: LEADER
I20260927 03:58:28.845796 18403 consensus_queue.cc:237] T 55c979b5d6e64481959e2e65e319fc9a P 1c62cb4ea2bf45a1830e7048550cf4db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a4ce643e7f40d4b3ff4892217c5352" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 45357 } } peers { permanent_uuid: "be26fcebf49a43259cadc0f2b9c0325a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38903 } } peers { permanent_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36419 } }
I20260927 03:58:28.846395 17828 catalog_manager.cc:5495] T 55c979b5d6e64481959e2e65e319fc9a P 1c62cb4ea2bf45a1830e7048550cf4db reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c62cb4ea2bf45a1830e7048550cf4db (127.15.185.67). New cstate: current_term: 1 leader_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47a4ce643e7f40d4b3ff4892217c5352" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 45357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be26fcebf49a43259cadc0f2b9c0325a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c62cb4ea2bf45a1830e7048550cf4db" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36419 } health_report { overall_health: HEALTHY } } }
I20260927 03:58:28.857472 18433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:28.855136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41708 (local address 127.15.185.67:36419)
0927 03:58:28.855180 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:28.855182 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:28.855191 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:58:28.855265 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:28.855266 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:28.855276 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:28.855330 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:58:28.855444 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:28.855933 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:28.856671 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:28.856960 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:28.857049 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:28.857211 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:58:28.857244 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:58:28.857295 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:28.857360 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 03:58:28.857396 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 03:58:28.859572 18125 raft_consensus.cc:1261] T 55c979b5d6e64481959e2e65e319fc9a P be26fcebf49a43259cadc0f2b9c0325a [term 1 FOLLOWER]: Refusing update from remote peer 1c62cb4ea2bf45a1830e7048550cf4db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:58:28.859617 17982 raft_consensus.cc:1261] T 55c979b5d6e64481959e2e65e319fc9a P 47a4ce643e7f40d4b3ff4892217c5352 [term 1 FOLLOWER]: Refusing update from remote peer 1c62cb4ea2bf45a1830e7048550cf4db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:58:28.859779 18407 consensus_queue.cc:1035] T 55c979b5d6e64481959e2e65e319fc9a P 1c62cb4ea2bf45a1830e7048550cf4db [LEADER]: Connected to new peer: Peer: permanent_uuid: "be26fcebf49a43259cadc0f2b9c0325a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:58:28.859856 18380 consensus_queue.cc:1035] T 55c979b5d6e64481959e2e65e319fc9a P 1c62cb4ea2bf45a1830e7048550cf4db [LEADER]: Connected to new peer: Peer: permanent_uuid: "47a4ce643e7f40d4b3ff4892217c5352" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 45357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 03:58:28.864147 18394 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 03:58:30.380873 17826 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:58:30.380975 17826 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:58:30.381798 17826 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 67a6d9bd1a4e47818b720084e11ccd8a: Sending DeleteTablet for 3 replicas of tablet 55c979b5d6e64481959e2e65e319fc9a
I20260927 03:58:30.382066 17965 tablet_service.cc:1514] Processing DeleteTablet for tablet 55c979b5d6e64481959e2e65e319fc9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:58:30 UTC) from {username='kudu', principal='kudu/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:35292
I20260927 03:58:30.382097 18105 tablet_service.cc:1514] Processing DeleteTablet for tablet 55c979b5d6e64481959e2e65e319fc9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:58:30 UTC) from {username='kudu', principal='kudu/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:56438
I20260927 03:58:30.382090 18245 tablet_service.cc:1514] Processing DeleteTablet for tablet 55c979b5d6e64481959e2e65e319fc9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:58:30 UTC) from {username='kudu', principal='kudu/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:41692
I20260927 03:58:30.382334 18479 tablet_replica.cc:330] T 55c979b5d6e64481959e2e65e319fc9a P 1c62cb4ea2bf45a1830e7048550cf4db: stopping tablet replica
I20260927 03:58:30.382409 18479 raft_consensus.cc:2229] T 55c979b5d6e64481959e2e65e319fc9a P 1c62cb4ea2bf45a1830e7048550cf4db [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:58:30.382424 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 17900
I20260927 03:58:30.382503 18479 raft_consensus.cc:2258] T 55c979b5d6e64481959e2e65e319fc9a P 1c62cb4ea2bf45a1830e7048550cf4db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:58:30.383307 18479 ts_tablet_manager.cc:1905] T 55c979b5d6e64481959e2e65e319fc9a P 1c62cb4ea2bf45a1830e7048550cf4db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:58:30.384817 18479 ts_tablet_manager.cc:1918] T 55c979b5d6e64481959e2e65e319fc9a P 1c62cb4ea2bf45a1830e7048550cf4db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 03:58:30.384863 18479 log.cc:1196] T 55c979b5d6e64481959e2e65e319fc9a P 1c62cb4ea2bf45a1830e7048550cf4db: Deleting WAL directory at /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/ts-2/wal/wals/55c979b5d6e64481959e2e65e319fc9a
I20260927 03:58:30.385093 18479 ts_tablet_manager.cc:1939] T 55c979b5d6e64481959e2e65e319fc9a P 1c62cb4ea2bf45a1830e7048550cf4db: Deleting consensus metadata
I20260927 03:58:30.385417 17812 catalog_manager.cc:4841] TS 1c62cb4ea2bf45a1830e7048550cf4db (127.15.185.67:36419): tablet 55c979b5d6e64481959e2e65e319fc9a (table test-table [id=774f977571b9484c8f1c21f4cda5962c]) successfully deleted
I20260927 03:58:30.387204 18478 tablet_replica.cc:330] T 55c979b5d6e64481959e2e65e319fc9a P be26fcebf49a43259cadc0f2b9c0325a: stopping tablet replica
I20260927 03:58:30.387288 18478 raft_consensus.cc:2229] T 55c979b5d6e64481959e2e65e319fc9a P be26fcebf49a43259cadc0f2b9c0325a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:58:30.387332 18478 raft_consensus.cc:2258] T 55c979b5d6e64481959e2e65e319fc9a P be26fcebf49a43259cadc0f2b9c0325a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:58:30.387579 18478 ts_tablet_manager.cc:1905] T 55c979b5d6e64481959e2e65e319fc9a P be26fcebf49a43259cadc0f2b9c0325a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:58:30.389564 18478 ts_tablet_manager.cc:1918] T 55c979b5d6e64481959e2e65e319fc9a P be26fcebf49a43259cadc0f2b9c0325a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 03:58:30.389628 18478 log.cc:1196] T 55c979b5d6e64481959e2e65e319fc9a P be26fcebf49a43259cadc0f2b9c0325a: Deleting WAL directory at /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481498018407-16101-0/minicluster-data/ts-1/wal/wals/55c979b5d6e64481959e2e65e319fc9a
I20260927 03:58:30.389853 18478 ts_tablet_manager.cc:1939] T 55c979b5d6e64481959e2e65e319fc9a P be26fcebf49a43259cadc0f2b9c0325a: Deleting consensus metadata
W20260927 03:58:30.390375 17814 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.185.65:45357 (error 108)
I20260927 03:58:30.390513 17815 catalog_manager.cc:4841] TS be26fcebf49a43259cadc0f2b9c0325a (127.15.185.66:38903): tablet 55c979b5d6e64481959e2e65e319fc9a (table test-table [id=774f977571b9484c8f1c21f4cda5962c]) successfully deleted
I20260927 03:58:30.391034 18481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:30.390780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.65:45357 (local address 127.0.0.1:44608)
0927 03:58:30.390917 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:30.390991 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.65:45357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":66,"threads_started":1}
I20260927 03:58:30.391077 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 18040
W20260927 03:58:30.391170 17814 catalog_manager.cc:4584] TS 47a4ce643e7f40d4b3ff4892217c5352 (127.15.185.65:45357): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 55c979b5d6e64481959e2e65e319fc9a: Network error: Client connection negotiation failed: client connection to 127.15.185.65:45357: connect: Connection refused (error 111)
I20260927 03:58:30.397223 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 18180
I20260927 03:58:30.402653 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 17796
2026-09-27T03:58:30Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5872 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:58:30 dist-test-slave-glgz krb5kdc[18484](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:58:30 dist-test-slave-glgz krb5kdc[18484](info): set up 2 sockets
Sep 27 03:58:30 dist-test-slave-glgz krb5kdc[18484](info): commencing operation
krb5kdc: starting...
W20260927 03:58:31.447335 16101 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Sep 27 03:58:31 dist-test-slave-glgz krb5kdc[18484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:58:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:58:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.185.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab.
Entry for principal kudu/127.15.185.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab.
I20260927 03:58:31.569911 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:46429
--webserver_interface=127.15.185.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:42973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.185.126:46429
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:58:31.659246 18497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:31.659540 18497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:31.659586 18497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:31.660923 18497 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:58:31.661007 18497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:31.661036 18497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:31.661072 18497 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:58:31.661096 18497 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:58:31.662825 18497 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:42973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.185.126:46429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:46429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18497
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:31.663187 18497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:31.663471 18497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 03:58:31.668828 18497 server_base.cc:1034] running on GCE node
W20260927 03:58:31.668798 18503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:31.668790 18502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:31.668808 18505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:31.669277 18497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:31.669596 18497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:31.670827 18497 hybrid_clock.cc:648] HybridClock initialized: now 1790481511670804 us; error 36 us; skew 500 ppm
Sep 27 03:58:31 dist-test-slave-glgz krb5kdc[18484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481511, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:31.673496 18497 init.cc:377] Logged in from keytab as kudu/127.15.185.126@KRBTEST.COM (short username kudu)
I20260927 03:58:31.673954 18497 webserver.cc:460] Webserver started at http://127.15.185.126:44447/ using document root <none> and password file <none>
I20260927 03:58:31.674196 18497 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:31.674265 18497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:31.674417 18497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:31.675611 18497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/minicluster-data/master-0/data/instance:
uuid: "1a0515c8e1cb4d7191e03e3713ac6086"
format_stamp: "Formatted at 2026-09-27 03:58:31 on dist-test-slave-glgz"
server_key: "aab50a47986dc7d1387f48d488462bf9"
server_key_iv: "e862a6902882462017cb74387aa9ac45"
server_key_version: "encryptionkey@0"
I20260927 03:58:31.675981 18497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/minicluster-data/master-0/wal/instance:
uuid: "1a0515c8e1cb4d7191e03e3713ac6086"
format_stamp: "Formatted at 2026-09-27 03:58:31 on dist-test-slave-glgz"
server_key: "aab50a47986dc7d1387f48d488462bf9"
server_key_iv: "e862a6902882462017cb74387aa9ac45"
server_key_version: "encryptionkey@0"
I20260927 03:58:31.677333 18497 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260927 03:58:31.678061 18512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:31.678237 18497 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:58:31.678344 18497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/minicluster-data/master-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/minicluster-data/master-0/wal
uuid: "1a0515c8e1cb4d7191e03e3713ac6086"
format_stamp: "Formatted at 2026-09-27 03:58:31 on dist-test-slave-glgz"
server_key: "aab50a47986dc7d1387f48d488462bf9"
server_key_iv: "e862a6902882462017cb74387aa9ac45"
server_key_version: "encryptionkey@0"
I20260927 03:58:31.678419 18497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:31.695343 18497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:31.697655 18497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:31.697821 18497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:31.701818 18497 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.126:46429
I20260927 03:58:31.701857 18564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.126:46429 every 8 connection(s)
I20260927 03:58:31.702243 18497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
I20260927 03:58:31.703215 18565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:31.705147 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 18497
I20260927 03:58:31.705255 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/minicluster-data/master-0/wal/instance
I20260927 03:58:31.705436 16101 external_mini_cluster.cc:1428] Setting key 809f206db247edfb125562fea26c01d3
I20260927 03:58:31.705772 18565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a0515c8e1cb4d7191e03e3713ac6086: Bootstrap starting.
I20260927 03:58:31.706445 18565 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a0515c8e1cb4d7191e03e3713ac6086: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:31.706789 18565 log.cc:824] T 00000000000000000000000000000000 P 1a0515c8e1cb4d7191e03e3713ac6086: Log is configured to *not* fsync() on all Append() calls
I20260927 03:58:31.707629 18565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a0515c8e1cb4d7191e03e3713ac6086: No bootstrap required, opened a new log
I20260927 03:58:31.709877 18565 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a0515c8e1cb4d7191e03e3713ac6086 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a0515c8e1cb4d7191e03e3713ac6086" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 46429 } }
I20260927 03:58:31.710093 18565 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a0515c8e1cb4d7191e03e3713ac6086 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:31.710124 18565 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a0515c8e1cb4d7191e03e3713ac6086 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a0515c8e1cb4d7191e03e3713ac6086, State: Initialized, Role: FOLLOWER
I20260927 03:58:31.710289 18565 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a0515c8e1cb4d7191e03e3713ac6086 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a0515c8e1cb4d7191e03e3713ac6086" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 46429 } }
I20260927 03:58:31.710354 18565 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a0515c8e1cb4d7191e03e3713ac6086 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:58:31.710377 18565 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a0515c8e1cb4d7191e03e3713ac6086 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:58:31.710448 18565 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a0515c8e1cb4d7191e03e3713ac6086 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:31.711311 18565 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a0515c8e1cb4d7191e03e3713ac6086 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a0515c8e1cb4d7191e03e3713ac6086" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 46429 } }
I20260927 03:58:31.711437 18565 leader_election.cc:302] T 00000000000000000000000000000000 P 1a0515c8e1cb4d7191e03e3713ac6086 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a0515c8e1cb4d7191e03e3713ac6086; no voters:
I20260927 03:58:31.711639 18565 leader_election.cc:288] T 00000000000000000000000000000000 P 1a0515c8e1cb4d7191e03e3713ac6086 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:58:31.711758 18570 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a0515c8e1cb4d7191e03e3713ac6086 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:58:31.711931 18570 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a0515c8e1cb4d7191e03e3713ac6086 [term 1 LEADER]: Becoming Leader. State: Replica: 1a0515c8e1cb4d7191e03e3713ac6086, State: Running, Role: LEADER
I20260927 03:58:31.711948 18565 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a0515c8e1cb4d7191e03e3713ac6086 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:58:31.712137 18570 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a0515c8e1cb4d7191e03e3713ac6086 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a0515c8e1cb4d7191e03e3713ac6086" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 46429 } }
I20260927 03:58:31.712502 18571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a0515c8e1cb4d7191e03e3713ac6086 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a0515c8e1cb4d7191e03e3713ac6086" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a0515c8e1cb4d7191e03e3713ac6086" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 46429 } } }
I20260927 03:58:31.712582 18571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a0515c8e1cb4d7191e03e3713ac6086 [sys.catalog]: This master's current role is: LEADER
I20260927 03:58:31.712738 18572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a0515c8e1cb4d7191e03e3713ac6086 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a0515c8e1cb4d7191e03e3713ac6086. Latest consensus state: current_term: 1 leader_uuid: "1a0515c8e1cb4d7191e03e3713ac6086" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a0515c8e1cb4d7191e03e3713ac6086" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 46429 } } }
I20260927 03:58:31.712807 18572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a0515c8e1cb4d7191e03e3713ac6086 [sys.catalog]: This master's current role is: LEADER
Sep 27 03:58:31 dist-test-slave-glgz krb5kdc[18484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
W20260927 03:58:31.714072 18586 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1a0515c8e1cb4d7191e03e3713ac6086: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:58:31.714136 18586 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:58:31.714179 18577 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:58:31.714326 18577 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:58:31.716001 18577 catalog_manager.cc:1347] Generated new cluster ID: 9a911038cd444e1680e7892081105908
I20260927 03:58:31.716055 18577 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:58:31.718014 18568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:31.705956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38266 (local address 127.15.185.126:46429)
0927 03:58:31.706147 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:31.706157 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:31.706190 (+ 33us) server_negotiation.cc:407] Connection header received
0927 03:58:31.706673 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:31.706680 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:31.706728 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:31.706896 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:31.707809 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:31.708707 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:31.709993 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:31.710613 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:31.710696 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:31.714104 (+ 3408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:31.714108 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:31.714108 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:31.714121 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:31.716333 (+ 2212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:31.716739 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:31.716742 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:31.716743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:31.716838 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:31.717062 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:31.717063 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:31.717064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:31.717322 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:31.717512 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:31.717629 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 03:58:31.717778 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":87,"threads_started":1}
I20260927 03:58:31.743187 18577 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:58:31.743922 18577 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:58:31.751976 18577 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a0515c8e1cb4d7191e03e3713ac6086: Generated new TSK 0
I20260927 03:58:31.752202 18577 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.185.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab.
Entry for principal kudu/127.15.185.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab.
I20260927 03:58:31.786952 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.65:0
--local_ip_for_outbound_sockets=127.15.185.65
--webserver_interface=127.15.185.65
--webserver_port=0
--tserver_master_addrs=127.15.185.126:46429
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab
--principal=kudu/127.15.185.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:42973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:58:31.870134 18591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:31.870347 18591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:31.870371 18591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:31.871645 18591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:31.871688 18591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:31.871714 18591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.65
I20260927 03:58:31.873093 18591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:42973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab
--principal=kudu/127.15.185.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.185.65
--webserver_port=0
--tserver_master_addrs=127.15.185.126:46429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18591
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.65
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:31.873281 18591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:31.873461 18591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:31.878710 18596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:31.878713 18597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:31.878988 18591 server_base.cc:1034] running on GCE node
W20260927 03:58:31.878742 18599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:31.879472 18591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:31.879706 18591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:31.880888 18591 hybrid_clock.cc:648] HybridClock initialized: now 1790481511880876 us; error 40 us; skew 500 ppm
Sep 27 03:58:31 dist-test-slave-glgz krb5kdc[18484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481511, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:31.883687 18591 init.cc:377] Logged in from keytab as kudu/127.15.185.65@KRBTEST.COM (short username kudu)
I20260927 03:58:31.884174 18591 webserver.cc:460] Webserver started at http://127.15.185.65:33303/ using document root <none> and password file <none>
I20260927 03:58:31.884409 18591 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:31.884479 18591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:31.884635 18591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:31.885710 18591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-0/data/instance:
uuid: "2edfdd21dfef4d619239a414f547484d"
format_stamp: "Formatted at 2026-09-27 03:58:31 on dist-test-slave-glgz"
server_key: "c26e082b575022d2f608f99d3a2cb05e"
server_key_iv: "26631f1905f69cb3e9a9cf247258940b"
server_key_version: "encryptionkey@0"
I20260927 03:58:31.886085 18591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-0/wal/instance:
uuid: "2edfdd21dfef4d619239a414f547484d"
format_stamp: "Formatted at 2026-09-27 03:58:31 on dist-test-slave-glgz"
server_key: "c26e082b575022d2f608f99d3a2cb05e"
server_key_iv: "26631f1905f69cb3e9a9cf247258940b"
server_key_version: "encryptionkey@0"
I20260927 03:58:31.887527 18591 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:58:31.888280 18606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:31.888443 18591 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:58:31.888545 18591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-0/wal
uuid: "2edfdd21dfef4d619239a414f547484d"
format_stamp: "Formatted at 2026-09-27 03:58:31 on dist-test-slave-glgz"
server_key: "c26e082b575022d2f608f99d3a2cb05e"
server_key_iv: "26631f1905f69cb3e9a9cf247258940b"
server_key_version: "encryptionkey@0"
I20260927 03:58:31.888628 18591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:31.898857 18591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:31.901535 18591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:31.901714 18591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:31.901971 18591 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:58:31.902297 18591 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:58:31.902352 18591 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:31.902393 18591 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:58:31.902421 18591 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:31.908363 18591 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.65:33351
I20260927 03:58:31.908442 18719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.65:33351 every 8 connection(s)
I20260927 03:58:31.908793 18591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
I20260927 03:58:31.912212 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 18591
I20260927 03:58:31.912336 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-0/wal/instance
I20260927 03:58:31.912513 16101 external_mini_cluster.cc:1428] Setting key e84422017d7a08f8dc22d3b710069a74
Sep 27 03:58:31 dist-test-slave-glgz krb5kdc[18484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481511, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.65@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 03:58:31.920007 18568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:31.909633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:57135 (local address 127.15.185.126:46429)
0927 03:58:31.909704 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:31.909707 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:31.909918 (+ 211us) server_negotiation.cc:407] Connection header received
0927 03:58:31.910405 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:31.910406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:31.910417 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:31.910486 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:31.911472 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:31.911998 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:31.913471 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:31.913828 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:31.913938 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:31.916668 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:31.916670 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:31.916670 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:31.916679 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:31.918332 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:31.918812 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:31.918813 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:31.918814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:31.918882 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:31.919269 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:31.919270 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:31.919271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:31.919406 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:31.919493 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:31.919797 (+ 304us) server_negotiation.cc:299] Negotiation successful
0927 03:58:31.919905 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 03:58:31.920222 18722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:31.909613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:46429 (local address 127.15.185.65:57135)
0927 03:58:31.909777 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:31.909822 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:31.910309 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:31.910563 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:31.910566 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:31.911178 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:31.911383 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:31.911387 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:31.912150 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:31.912151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:31.913387 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:31.913390 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:31.913958 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:31.913959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:31.914071 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:31.914698 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:31.914710 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:31.916537 (+ 1827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:31.918454 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:31.918462 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:31.918465 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:31.918724 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:31.918968 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:31.918969 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:31.918970 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:31.919192 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:31.919484 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:31.919492 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:31.919649 (+ 157us) client_negotiation.cc:769] Sending connection context
0927 03:58:31.919762 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 03:58:31.919939 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":93,"threads_started":1}
WARNING: no policy specified for kudu/127.15.185.66@KRBTEST.COM; defaulting to no policy
I20260927 03:58:31.920892 18720 heartbeater.cc:347] Connected to a master server at 127.15.185.126:46429
I20260927 03:58:31.921002 18720 heartbeater.cc:464] Registering TS with master...
I20260927 03:58:31.921171 18720 heartbeater.cc:511] Master 127.15.185.126:46429 requested a full tablet report, sending...
I20260927 03:58:31.921723 18529 ts_manager.cc:194] Registered new tserver with Master: 2edfdd21dfef4d619239a414f547484d (127.15.185.65:33351)
I20260927 03:58:31.922377 18529 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.185.65@KRBTEST.COM'} at 127.15.185.65:57135
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab.
Entry for principal kudu/127.15.185.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab.
I20260927 03:58:31.940256 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.66:0
--local_ip_for_outbound_sockets=127.15.185.66
--webserver_interface=127.15.185.66
--webserver_port=0
--tserver_master_addrs=127.15.185.126:46429
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab
--principal=kudu/127.15.185.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:42973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:58:32.023969 18725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:32.024242 18725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:32.024282 18725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:32.025494 18725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:32.025574 18725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:32.025621 18725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.66
I20260927 03:58:32.027144 18725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:42973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab
--principal=kudu/127.15.185.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.185.66
--webserver_port=0
--tserver_master_addrs=127.15.185.126:46429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.66
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:32.027433 18725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:32.027690 18725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:32.032932 18733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:32.032984 18731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:32.032984 18730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:32.033321 18725 server_base.cc:1034] running on GCE node
I20260927 03:58:32.033588 18725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:32.033886 18725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:32.035128 18725 hybrid_clock.cc:648] HybridClock initialized: now 1790481512035114 us; error 30 us; skew 500 ppm
Sep 27 03:58:32 dist-test-slave-glgz krb5kdc[18484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481512, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:32.037627 18725 init.cc:377] Logged in from keytab as kudu/127.15.185.66@KRBTEST.COM (short username kudu)
I20260927 03:58:32.038082 18725 webserver.cc:460] Webserver started at http://127.15.185.66:37351/ using document root <none> and password file <none>
I20260927 03:58:32.038314 18725 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:32.038383 18725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:32.038533 18725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:32.039678 18725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-1/data/instance:
uuid: "51524a82e4f2475c9aa3b8935890e410"
format_stamp: "Formatted at 2026-09-27 03:58:32 on dist-test-slave-glgz"
server_key: "c05ac893a6fe7faed31a4264cdaa211b"
server_key_iv: "1a24755d1bc904e708123af667426bb2"
server_key_version: "encryptionkey@0"
I20260927 03:58:32.040036 18725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-1/wal/instance:
uuid: "51524a82e4f2475c9aa3b8935890e410"
format_stamp: "Formatted at 2026-09-27 03:58:32 on dist-test-slave-glgz"
server_key: "c05ac893a6fe7faed31a4264cdaa211b"
server_key_iv: "1a24755d1bc904e708123af667426bb2"
server_key_version: "encryptionkey@0"
I20260927 03:58:32.041381 18725 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:58:32.042089 18740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:32.042258 18725 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:58:32.042356 18725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-1/wal
uuid: "51524a82e4f2475c9aa3b8935890e410"
format_stamp: "Formatted at 2026-09-27 03:58:32 on dist-test-slave-glgz"
server_key: "c05ac893a6fe7faed31a4264cdaa211b"
server_key_iv: "1a24755d1bc904e708123af667426bb2"
server_key_version: "encryptionkey@0"
I20260927 03:58:32.042433 18725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:32.054610 18725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:32.057292 18725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:32.057484 18725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:32.057746 18725 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:58:32.058085 18725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:58:32.058139 18725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:32.058174 18725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:58:32.058208 18725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:32.064312 18725 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.66:41941
I20260927 03:58:32.064363 18853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.66:41941 every 8 connection(s)
I20260927 03:58:32.064729 18725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
I20260927 03:58:32.065722 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 18725
I20260927 03:58:32.065832 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-1/wal/instance
I20260927 03:58:32.066051 16101 external_mini_cluster.cc:1428] Setting key ea70e2b98cd45584f930684ee7800b31
Sep 27 03:58:32 dist-test-slave-glgz krb5kdc[18484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481512, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.66@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.185.67@KRBTEST.COM; defaulting to no policy
I20260927 03:58:32.075248 18568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:32.065660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:58655 (local address 127.15.185.126:46429)
0927 03:58:32.065713 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:32.065715 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:32.065916 (+ 201us) server_negotiation.cc:407] Connection header received
0927 03:58:32.066563 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:32.066564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:32.066576 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:32.066623 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:32.067671 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:32.068178 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:32.069191 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:32.069396 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:32.069484 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:32.072078 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:32.072080 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:32.072080 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:32.072088 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:32.073585 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:32.074137 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:32.074139 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:32.074139 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:32.074201 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:32.074456 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:32.074457 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:32.074457 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:32.074575 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:32.074654 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:32.075037 (+ 383us) server_negotiation.cc:299] Negotiation successful
0927 03:58:32.075172 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 03:58:32.075410 18856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:32.065550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:46429 (local address 127.15.185.66:58655)
0927 03:58:32.065764 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:32.065821 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:32.066451 (+ 630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:32.066693 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:32.066697 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:32.067362 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:32.067563 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:32.067569 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:32.068275 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:32.068276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:32.069083 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:32.069085 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:32.069494 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:32.069495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:32.069620 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:32.070261 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:32.070270 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:32.071969 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:32.073726 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:32.073732 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:32.073735 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:32.074046 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:32.074268 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:32.074269 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:32.074269 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:32.074386 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:32.074649 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:32.074656 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:32.074819 (+ 163us) client_negotiation.cc:769] Sending connection context
0927 03:58:32.074992 (+ 173us) client_negotiation.cc:240] Negotiation successful
0927 03:58:32.075185 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":107,"threads_started":1}
I20260927 03:58:32.075930 18854 heartbeater.cc:347] Connected to a master server at 127.15.185.126:46429
I20260927 03:58:32.076052 18854 heartbeater.cc:464] Registering TS with master...
I20260927 03:58:32.076239 18854 heartbeater.cc:511] Master 127.15.185.126:46429 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.67@KRBTEST.COM" created.
I20260927 03:58:32.076684 18529 ts_manager.cc:194] Registered new tserver with Master: 51524a82e4f2475c9aa3b8935890e410 (127.15.185.66:41941)
I20260927 03:58:32.077116 18529 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.185.66@KRBTEST.COM'} at 127.15.185.66:58655
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab.
Entry for principal kudu/127.15.185.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab.
I20260927 03:58:32.093865 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.67:0
--local_ip_for_outbound_sockets=127.15.185.67
--webserver_interface=127.15.185.67
--webserver_port=0
--tserver_master_addrs=127.15.185.126:46429
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab
--principal=kudu/127.15.185.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:42973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:58:32.176363 18859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:32.176618 18859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:32.176644 18859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:32.177860 18859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:32.177913 18859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:32.177948 18859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.67
I20260927 03:58:32.179543 18859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:42973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab
--principal=kudu/127.15.185.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.185.67
--webserver_port=0
--tserver_master_addrs=127.15.185.126:46429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.67
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:32.179831 18859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:32.180087 18859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:32.185618 18864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:32.185611 18867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:32.185608 18865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:32.185775 18859 server_base.cc:1034] running on GCE node
I20260927 03:58:32.186144 18859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:32.186417 18859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:32.187584 18859 hybrid_clock.cc:648] HybridClock initialized: now 1790481512187573 us; error 47 us; skew 500 ppm
Sep 27 03:58:32 dist-test-slave-glgz krb5kdc[18484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481512, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:32.190246 18859 init.cc:377] Logged in from keytab as kudu/127.15.185.67@KRBTEST.COM (short username kudu)
I20260927 03:58:32.190757 18859 webserver.cc:460] Webserver started at http://127.15.185.67:45783/ using document root <none> and password file <none>
I20260927 03:58:32.191006 18859 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:32.191099 18859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:32.191248 18859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:32.192380 18859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-2/data/instance:
uuid: "b05c92c4fbed4223b5a164f8c11bc05f"
format_stamp: "Formatted at 2026-09-27 03:58:32 on dist-test-slave-glgz"
server_key: "872b3ba6ceecdadfa1ba55387755d29e"
server_key_iv: "cc328c235f4d8693c6e6f9361375903c"
server_key_version: "encryptionkey@0"
I20260927 03:58:32.192760 18859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-2/wal/instance:
uuid: "b05c92c4fbed4223b5a164f8c11bc05f"
format_stamp: "Formatted at 2026-09-27 03:58:32 on dist-test-slave-glgz"
server_key: "872b3ba6ceecdadfa1ba55387755d29e"
server_key_iv: "cc328c235f4d8693c6e6f9361375903c"
server_key_version: "encryptionkey@0"
I20260927 03:58:32.194238 18859 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:58:32.195215 18874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:32.195437 18859 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:58:32.195561 18859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-2/wal
uuid: "b05c92c4fbed4223b5a164f8c11bc05f"
format_stamp: "Formatted at 2026-09-27 03:58:32 on dist-test-slave-glgz"
server_key: "872b3ba6ceecdadfa1ba55387755d29e"
server_key_iv: "cc328c235f4d8693c6e6f9361375903c"
server_key_version: "encryptionkey@0"
I20260927 03:58:32.195654 18859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:32.216724 18859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:32.219553 18859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:32.219750 18859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:32.220031 18859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:58:32.220383 18859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:58:32.220451 18859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:32.220490 18859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:58:32.220527 18859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:32.227360 18859 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.67:38533
I20260927 03:58:32.227406 18987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.67:38533 every 8 connection(s)
I20260927 03:58:32.227823 18859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
I20260927 03:58:32.229163 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 18859
I20260927 03:58:32.229269 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-2/wal/instance
I20260927 03:58:32.229460 16101 external_mini_cluster.cc:1428] Setting key ad01118ce4c6f0f58b907f125d7ff8b4
Sep 27 03:58:32 dist-test-slave-glgz krb5kdc[18484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481512, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.67@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 03:58:32.239490 18568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:32.228709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:44773 (local address 127.15.185.126:46429)
0927 03:58:32.228801 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:32.228803 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:32.229032 (+ 229us) server_negotiation.cc:407] Connection header received
0927 03:58:32.229806 (+ 774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:32.229808 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:32.229820 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:32.229873 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:32.231241 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:32.231750 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:32.232907 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:32.233162 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:32.233238 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:32.235691 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:32.235693 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:32.235694 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:32.235704 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:32.237771 (+ 2067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:32.238332 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:32.238334 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:32.238334 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:32.238402 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:32.238692 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:32.238693 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:32.238694 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:32.238823 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:32.238901 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:32.239297 (+ 396us) server_negotiation.cc:299] Negotiation successful
0927 03:58:32.239383 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 03:58:32.239696 18990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:32.228664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:46429 (local address 127.15.185.67:44773)
0927 03:58:32.228846 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:32.228896 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:32.229715 (+ 819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:32.229972 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:32.229984 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:32.230840 (+ 856us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:32.231128 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:32.231136 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:32.231841 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:32.231842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:32.232790 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:32.232793 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:32.233257 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:32.233258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:32.233380 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:32.233998 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:32.234010 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:32.235572 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:32.237881 (+ 2309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:32.237890 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:32.237893 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:32.238229 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:32.238488 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:32.238489 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:32.238490 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:32.238621 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:32.238892 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:32.238898 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:32.239113 (+ 215us) client_negotiation.cc:769] Sending connection context
0927 03:58:32.239248 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 03:58:32.239421 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":96,"threads_started":1}
I20260927 03:58:32.240239 18988 heartbeater.cc:347] Connected to a master server at 127.15.185.126:46429
I20260927 03:58:32.240345 18988 heartbeater.cc:464] Registering TS with master...
I20260927 03:58:32.240554 18988 heartbeater.cc:511] Master 127.15.185.126:46429 requested a full tablet report, sending...
I20260927 03:58:32.240957 18528 ts_manager.cc:194] Registered new tserver with Master: b05c92c4fbed4223b5a164f8c11bc05f (127.15.185.67:38533)
I20260927 03:58:32.241465 18528 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.185.67@KRBTEST.COM'} at 127.15.185.67:44773
I20260927 03:58:32.241544 16101 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 03:58:32.253575 18568 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:58:32.252543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38272 (local address 127.15.185.126:46429)
0927 03:58:32.252615 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:32.252618 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:32.252729 (+ 111us) server_negotiation.cc:407] Connection header received
0927 03:58:32.252834 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:32.252835 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:32.252846 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:32.252897 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:32.253493 (+ 596us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38272: BlockingRecv error: recv got EOF from 127.0.0.1:38272 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 03:58:32.254432 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 18591
I20260927 03:58:32.259743 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 18725
I20260927 03:58:32.264554 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 18859
I20260927 03:58:32.269215 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 18497
2026-09-27T03:58:32Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1864 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481498018407-16101-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:58:32 dist-test-slave-glgz krb5kdc[19000](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:58:32 dist-test-slave-glgz krb5kdc[19000](info): set up 2 sockets
Sep 27 03:58:32 dist-test-slave-glgz krb5kdc[19000](info): commencing operation
krb5kdc: starting...
W20260927 03:58:33.319507 16101 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Sep 27 03:58:33 dist-test-slave-glgz krb5kdc[19000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:58:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:58:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.185.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab.
Entry for principal kudu/127.15.185.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab.
I20260927 03:58:33.437850 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481498018407-16101-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481498018407-16101-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481498018407-16101-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:45527
--webserver_interface=127.15.185.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:39087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.185.126:45527
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:58:33.518152 19013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:33.518365 19013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:33.518390 19013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:33.519508 19013 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:58:33.519542 19013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:33.519556 19013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:33.519569 19013 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:58:33.519582 19013 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:58:33.520984 19013 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:39087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481498018407-16101-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481498018407-16101-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.185.126:45527
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:45527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481498018407-16101-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:33.521166 19013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:33.521338 19013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:33.526166 19019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:33.526160 19018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:33.526139 19021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:33.526388 19013 server_base.cc:1034] running on GCE node
I20260927 03:58:33.526675 19013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:33.526962 19013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:33.528136 19013 hybrid_clock.cc:648] HybridClock initialized: now 1790481513528118 us; error 31 us; skew 500 ppm
Sep 27 03:58:33 dist-test-slave-glgz krb5kdc[19000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481513, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:33.537575 19013 init.cc:377] Logged in from keytab as kudu/127.15.185.126@KRBTEST.COM (short username kudu)
I20260927 03:58:33.538120 19013 webserver.cc:460] Webserver started at http://127.15.185.126:45233/ using document root <none> and password file <none>
I20260927 03:58:33.538363 19013 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:33.538432 19013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:33.538590 19013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:33.539726 19013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481498018407-16101-0/minicluster-data/master-0/data/instance:
uuid: "886727aef2cd4f77aed6bbe2e4884961"
format_stamp: "Formatted at 2026-09-27 03:58:33 on dist-test-slave-glgz"
server_key: "dc9e570ba6625aa896b5821119bca1e7"
server_key_iv: "9c0fae88577fa7193a50214a7c27f429"
server_key_version: "encryptionkey@0"
I20260927 03:58:33.540091 19013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481498018407-16101-0/minicluster-data/master-0/wal/instance:
uuid: "886727aef2cd4f77aed6bbe2e4884961"
format_stamp: "Formatted at 2026-09-27 03:58:33 on dist-test-slave-glgz"
server_key: "dc9e570ba6625aa896b5821119bca1e7"
server_key_iv: "9c0fae88577fa7193a50214a7c27f429"
server_key_version: "encryptionkey@0"
I20260927 03:58:33.541401 19013 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:58:33.542150 19028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:33.542331 19013 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:58:33.542436 19013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481498018407-16101-0/minicluster-data/master-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481498018407-16101-0/minicluster-data/master-0/wal
uuid: "886727aef2cd4f77aed6bbe2e4884961"
format_stamp: "Formatted at 2026-09-27 03:58:33 on dist-test-slave-glgz"
server_key: "dc9e570ba6625aa896b5821119bca1e7"
server_key_iv: "9c0fae88577fa7193a50214a7c27f429"
server_key_version: "encryptionkey@0"
I20260927 03:58:33.542513 19013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481498018407-16101-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481498018407-16101-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481498018407-16101-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:33.553071 19013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:33.555022 19013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:33.555226 19013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:33.559165 19013 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.126:45527
I20260927 03:58:33.559228 19080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.126:45527 every 8 connection(s)
I20260927 03:58:33.559705 19013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
I20260927 03:58:33.560316 19081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:33.562531 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 19013
I20260927 03:58:33.562538 19081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 886727aef2cd4f77aed6bbe2e4884961: Bootstrap starting.
I20260927 03:58:33.562629 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481498018407-16101-0/minicluster-data/master-0/wal/instance
I20260927 03:58:33.562816 16101 external_mini_cluster.cc:1428] Setting key f6b47d218c487082bc9fa83b33968bcd
I20260927 03:58:33.563341 19081 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 886727aef2cd4f77aed6bbe2e4884961: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:33.563659 19081 log.cc:824] T 00000000000000000000000000000000 P 886727aef2cd4f77aed6bbe2e4884961: Log is configured to *not* fsync() on all Append() calls
I20260927 03:58:33.564410 19081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 886727aef2cd4f77aed6bbe2e4884961: No bootstrap required, opened a new log
I20260927 03:58:33.566694 19081 raft_consensus.cc:348] T 00000000000000000000000000000000 P 886727aef2cd4f77aed6bbe2e4884961 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "886727aef2cd4f77aed6bbe2e4884961" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45527 } }
I20260927 03:58:33.566864 19081 raft_consensus.cc:374] T 00000000000000000000000000000000 P 886727aef2cd4f77aed6bbe2e4884961 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:33.566895 19081 raft_consensus.cc:729] T 00000000000000000000000000000000 P 886727aef2cd4f77aed6bbe2e4884961 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 886727aef2cd4f77aed6bbe2e4884961, State: Initialized, Role: FOLLOWER
I20260927 03:58:33.567011 19081 consensus_queue.cc:260] T 00000000000000000000000000000000 P 886727aef2cd4f77aed6bbe2e4884961 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "886727aef2cd4f77aed6bbe2e4884961" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45527 } }
I20260927 03:58:33.567112 19081 raft_consensus.cc:388] T 00000000000000000000000000000000 P 886727aef2cd4f77aed6bbe2e4884961 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:58:33.567199 19081 raft_consensus.cc:482] T 00000000000000000000000000000000 P 886727aef2cd4f77aed6bbe2e4884961 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:58:33.567276 19081 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 886727aef2cd4f77aed6bbe2e4884961 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:33.567940 19081 raft_consensus.cc:504] T 00000000000000000000000000000000 P 886727aef2cd4f77aed6bbe2e4884961 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "886727aef2cd4f77aed6bbe2e4884961" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45527 } }
I20260927 03:58:33.568064 19081 leader_election.cc:302] T 00000000000000000000000000000000 P 886727aef2cd4f77aed6bbe2e4884961 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 886727aef2cd4f77aed6bbe2e4884961; no voters:
I20260927 03:58:33.568265 19081 leader_election.cc:288] T 00000000000000000000000000000000 P 886727aef2cd4f77aed6bbe2e4884961 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:58:33.568434 19086 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 886727aef2cd4f77aed6bbe2e4884961 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:58:33.568537 19081 sys_catalog.cc:564] T 00000000000000000000000000000000 P 886727aef2cd4f77aed6bbe2e4884961 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:58:33.568602 19086 raft_consensus.cc:686] T 00000000000000000000000000000000 P 886727aef2cd4f77aed6bbe2e4884961 [term 1 LEADER]: Becoming Leader. State: Replica: 886727aef2cd4f77aed6bbe2e4884961, State: Running, Role: LEADER
I20260927 03:58:33.568696 19086 consensus_queue.cc:237] T 00000000000000000000000000000000 P 886727aef2cd4f77aed6bbe2e4884961 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "886727aef2cd4f77aed6bbe2e4884961" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45527 } }
I20260927 03:58:33.568995 19086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 886727aef2cd4f77aed6bbe2e4884961 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 886727aef2cd4f77aed6bbe2e4884961. Latest consensus state: current_term: 1 leader_uuid: "886727aef2cd4f77aed6bbe2e4884961" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "886727aef2cd4f77aed6bbe2e4884961" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45527 } } }
I20260927 03:58:33.569088 19086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 886727aef2cd4f77aed6bbe2e4884961 [sys.catalog]: This master's current role is: LEADER
I20260927 03:58:33.569005 19087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 886727aef2cd4f77aed6bbe2e4884961 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "886727aef2cd4f77aed6bbe2e4884961" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "886727aef2cd4f77aed6bbe2e4884961" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45527 } } }
I20260927 03:58:33.569228 19087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 886727aef2cd4f77aed6bbe2e4884961 [sys.catalog]: This master's current role is: LEADER
I20260927 03:58:33.569388 19093 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:58:33.569945 19093 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 03:58:33 dist-test-slave-glgz krb5kdc[19000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 03:58:33.571913 19093 catalog_manager.cc:1347] Generated new cluster ID: e25a10c0deb74b34908840aab3682c38
I20260927 03:58:33.571967 19093 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:58:33.574402 19084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:33.563367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45520 (local address 127.15.185.126:45527)
0927 03:58:33.563538 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:33.563547 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:33.563583 (+ 36us) server_negotiation.cc:407] Connection header received
0927 03:58:33.564181 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:33.564189 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:33.564237 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:33.564352 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:33.565177 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:33.566037 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:33.567163 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:33.567688 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:33.567770 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:33.570925 (+ 3155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:33.570929 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:33.570929 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:33.570941 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:33.572799 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:33.573191 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:33.573194 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:33.573194 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:33.573290 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:33.573517 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:33.573518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:33.573518 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:33.573758 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:33.573950 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:33.574036 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 03:58:33.574176 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":78,"threads_started":1}
I20260927 03:58:33.575323 16101 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:58:33.597146 19084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:33.592810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45536 (local address 127.15.185.126:45527)
0927 03:58:33.592856 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:33.592858 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:33.592976 (+ 118us) server_negotiation.cc:407] Connection header received
0927 03:58:33.593030 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:33.593031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:33.593042 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:33.593120 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:33.593541 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:33.594377 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:33.595409 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:33.595742 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:33.595818 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:33.596567 (+ 749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:33.596569 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:33.596569 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:58:33.596835 (+ 266us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:33.596855 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:58:33.596863 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:33.596937 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:33.597025 (+ 88us) server_negotiation.cc:299] Negotiation successful
0927 03:58:33.597057 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 03:58:33.598443 19093 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:58:33.598935 19093 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:58:33.605643 19093 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 886727aef2cd4f77aed6bbe2e4884961: Generated new TSK 0
I20260927 03:58:33.605865 19093 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:58:33.609145 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 19013
2026-09-27T03:58:33Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1337 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:58:33 dist-test-slave-glgz krb5kdc[19114](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:58:33 dist-test-slave-glgz krb5kdc[19114](info): set up 2 sockets
Sep 27 03:58:33 dist-test-slave-glgz krb5kdc[19114](info): commencing operation
krb5kdc: starting...
W20260927 03:58:34.643337 16101 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Sep 27 03:58:34 dist-test-slave-glgz krb5kdc[19114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:58:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:58:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.185.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab.
Entry for principal kudu/127.15.185.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab.
I20260927 03:58:34.767030 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:44607
--webserver_interface=127.15.185.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:43089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.185.126:44607
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:58:34.848743 19127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:34.849007 19127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:34.849046 19127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:34.850208 19127 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:58:34.850276 19127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:34.850301 19127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:34.850323 19127 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:58:34.850353 19127 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:58:34.851972 19127 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:43089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.185.126:44607
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:44607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:34.852257 19127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:34.852491 19127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:34.857473 19132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:34.857496 19133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:34.857473 19135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:34.857651 19127 server_base.cc:1034] running on GCE node
I20260927 03:58:34.857981 19127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:34.858289 19127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:34.859455 19127 hybrid_clock.cc:648] HybridClock initialized: now 1790481514859442 us; error 49 us; skew 500 ppm
Sep 27 03:58:34 dist-test-slave-glgz krb5kdc[19114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481514, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:34.862083 19127 init.cc:377] Logged in from keytab as kudu/127.15.185.126@KRBTEST.COM (short username kudu)
I20260927 03:58:34.862546 19127 webserver.cc:460] Webserver started at http://127.15.185.126:34965/ using document root <none> and password file <none>
I20260927 03:58:34.862741 19127 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:34.862782 19127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:34.862877 19127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:34.864035 19127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/minicluster-data/master-0/data/instance:
uuid: "c0d631ef7bdc45b18b5fc70a450005ac"
format_stamp: "Formatted at 2026-09-27 03:58:34 on dist-test-slave-glgz"
server_key: "3a827f4f348a8c4ab2324ae31070de0b"
server_key_iv: "ab39c4cc616d60c77aa8bbaaab24175d"
server_key_version: "encryptionkey@0"
I20260927 03:58:34.864333 19127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/minicluster-data/master-0/wal/instance:
uuid: "c0d631ef7bdc45b18b5fc70a450005ac"
format_stamp: "Formatted at 2026-09-27 03:58:34 on dist-test-slave-glgz"
server_key: "3a827f4f348a8c4ab2324ae31070de0b"
server_key_iv: "ab39c4cc616d60c77aa8bbaaab24175d"
server_key_version: "encryptionkey@0"
I20260927 03:58:34.865550 19127 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:58:34.866339 19142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:34.866556 19127 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:58:34.866654 19127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/minicluster-data/master-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/minicluster-data/master-0/wal
uuid: "c0d631ef7bdc45b18b5fc70a450005ac"
format_stamp: "Formatted at 2026-09-27 03:58:34 on dist-test-slave-glgz"
server_key: "3a827f4f348a8c4ab2324ae31070de0b"
server_key_iv: "ab39c4cc616d60c77aa8bbaaab24175d"
server_key_version: "encryptionkey@0"
I20260927 03:58:34.866715 19127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:34.878141 19127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:34.880169 19127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:34.880285 19127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:34.884344 19127 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.126:44607
I20260927 03:58:34.884418 19194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.126:44607 every 8 connection(s)
I20260927 03:58:34.884783 19127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
I20260927 03:58:34.885753 19195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:34.887995 19195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0d631ef7bdc45b18b5fc70a450005ac: Bootstrap starting.
I20260927 03:58:34.888582 19195 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0d631ef7bdc45b18b5fc70a450005ac: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:34.888863 19195 log.cc:824] T 00000000000000000000000000000000 P c0d631ef7bdc45b18b5fc70a450005ac: Log is configured to *not* fsync() on all Append() calls
I20260927 03:58:34.889586 19195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0d631ef7bdc45b18b5fc70a450005ac: No bootstrap required, opened a new log
I20260927 03:58:34.891177 19195 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0d631ef7bdc45b18b5fc70a450005ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d631ef7bdc45b18b5fc70a450005ac" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 44607 } }
I20260927 03:58:34.891331 19195 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0d631ef7bdc45b18b5fc70a450005ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:34.891376 19195 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0d631ef7bdc45b18b5fc70a450005ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0d631ef7bdc45b18b5fc70a450005ac, State: Initialized, Role: FOLLOWER
I20260927 03:58:34.891532 19195 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0d631ef7bdc45b18b5fc70a450005ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d631ef7bdc45b18b5fc70a450005ac" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 44607 } }
I20260927 03:58:34.891614 19195 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0d631ef7bdc45b18b5fc70a450005ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:58:34.891662 19195 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0d631ef7bdc45b18b5fc70a450005ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:58:34.891716 19195 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0d631ef7bdc45b18b5fc70a450005ac [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:34.892259 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 19127
I20260927 03:58:34.892402 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/minicluster-data/master-0/wal/instance
I20260927 03:58:34.892596 16101 external_mini_cluster.cc:1428] Setting key 10a855651ea0a660981860c93a5af421
I20260927 03:58:34.892765 19195 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0d631ef7bdc45b18b5fc70a450005ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d631ef7bdc45b18b5fc70a450005ac" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 44607 } }
I20260927 03:58:34.892889 19195 leader_election.cc:302] T 00000000000000000000000000000000 P c0d631ef7bdc45b18b5fc70a450005ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0d631ef7bdc45b18b5fc70a450005ac; no voters:
I20260927 03:58:34.893043 19195 leader_election.cc:288] T 00000000000000000000000000000000 P c0d631ef7bdc45b18b5fc70a450005ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:58:34.893249 19198 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0d631ef7bdc45b18b5fc70a450005ac [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:58:34.893294 19195 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0d631ef7bdc45b18b5fc70a450005ac [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:58:34.893360 19198 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0d631ef7bdc45b18b5fc70a450005ac [term 1 LEADER]: Becoming Leader. State: Replica: c0d631ef7bdc45b18b5fc70a450005ac, State: Running, Role: LEADER
I20260927 03:58:34.893442 19198 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0d631ef7bdc45b18b5fc70a450005ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d631ef7bdc45b18b5fc70a450005ac" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 44607 } }
I20260927 03:58:34.893826 19202 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0d631ef7bdc45b18b5fc70a450005ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0d631ef7bdc45b18b5fc70a450005ac. Latest consensus state: current_term: 1 leader_uuid: "c0d631ef7bdc45b18b5fc70a450005ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d631ef7bdc45b18b5fc70a450005ac" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 44607 } } }
I20260927 03:58:34.893848 19200 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0d631ef7bdc45b18b5fc70a450005ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0d631ef7bdc45b18b5fc70a450005ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0d631ef7bdc45b18b5fc70a450005ac" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 44607 } } }
I20260927 03:58:34.893963 19200 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0d631ef7bdc45b18b5fc70a450005ac [sys.catalog]: This master's current role is: LEADER
I20260927 03:58:34.894028 19202 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0d631ef7bdc45b18b5fc70a450005ac [sys.catalog]: This master's current role is: LEADER
W20260927 03:58:34.895164 19215 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c0d631ef7bdc45b18b5fc70a450005ac: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:58:34.895223 19215 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:58:34.895430 19209 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:58:34.895622 19209 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:58:34.897029 19209 catalog_manager.cc:1347] Generated new cluster ID: 5fa37af27a334c129b37b9bc567b1472
I20260927 03:58:34.897083 19209 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 03:58:34 dist-test-slave-glgz krb5kdc[19114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 03:58:34.905555 19201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:34.893133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54638 (local address 127.15.185.126:44607)
0927 03:58:34.893311 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:34.893322 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:34.893361 (+ 39us) server_negotiation.cc:407] Connection header received
0927 03:58:34.893797 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:34.893805 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:34.893883 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:34.894033 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:34.895011 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:34.896620 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:34.897703 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:34.898323 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:34.898447 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:34.901231 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:34.901235 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:34.901235 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:34.901248 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:34.903840 (+ 2592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:34.904272 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:34.904275 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:34.904276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:34.904391 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:34.904609 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:34.904610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:34.904611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:34.904906 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:34.905060 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:34.905178 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 03:58:34.905326 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":72,"threads_started":1}
I20260927 03:58:34.905755 19209 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:58:34.906378 19209 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:58:34.917884 19209 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0d631ef7bdc45b18b5fc70a450005ac: Generated new TSK 0
I20260927 03:58:34.918099 19209 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.185.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab.
Entry for principal kudu/127.15.185.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab.
I20260927 03:58:34.949545 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.65:0
--local_ip_for_outbound_sockets=127.15.185.65
--webserver_interface=127.15.185.65
--webserver_port=0
--tserver_master_addrs=127.15.185.126:44607
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab
--principal=kudu/127.15.185.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:43089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:58:35.044524 19221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:35.044786 19221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:35.044831 19221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:35.046046 19221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:35.046090 19221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:35.046116 19221 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.65
I20260927 03:58:35.047593 19221 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:43089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab
--principal=kudu/127.15.185.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.185.65
--webserver_port=0
--tserver_master_addrs=127.15.185.126:44607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.65
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:35.047833 19221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:35.048090 19221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:35.053272 19227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:35.053263 19229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:35.053274 19226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:35.053566 19221 server_base.cc:1034] running on GCE node
I20260927 03:58:35.053784 19221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:35.054067 19221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:35.055223 19221 hybrid_clock.cc:648] HybridClock initialized: now 1790481515055220 us; error 39 us; skew 500 ppm
Sep 27 03:58:35 dist-test-slave-glgz krb5kdc[19114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481515, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:35.057672 19221 init.cc:377] Logged in from keytab as kudu/127.15.185.65@KRBTEST.COM (short username kudu)
I20260927 03:58:35.058187 19221 webserver.cc:460] Webserver started at http://127.15.185.65:46111/ using document root <none> and password file <none>
I20260927 03:58:35.058424 19221 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:35.058490 19221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:35.058645 19221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:35.059798 19221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-0/data/instance:
uuid: "d9f06c89fe8a46dfb67afc3ceaa940eb"
format_stamp: "Formatted at 2026-09-27 03:58:35 on dist-test-slave-glgz"
server_key: "fb417cf55c9f0442670c5ec64ed5842b"
server_key_iv: "5b86ba9c27ddafaf18118b9bce9ae2af"
server_key_version: "encryptionkey@0"
I20260927 03:58:35.060170 19221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-0/wal/instance:
uuid: "d9f06c89fe8a46dfb67afc3ceaa940eb"
format_stamp: "Formatted at 2026-09-27 03:58:35 on dist-test-slave-glgz"
server_key: "fb417cf55c9f0442670c5ec64ed5842b"
server_key_iv: "5b86ba9c27ddafaf18118b9bce9ae2af"
server_key_version: "encryptionkey@0"
I20260927 03:58:35.061528 19221 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:58:35.062269 19236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:35.062446 19221 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:58:35.062547 19221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-0/wal
uuid: "d9f06c89fe8a46dfb67afc3ceaa940eb"
format_stamp: "Formatted at 2026-09-27 03:58:35 on dist-test-slave-glgz"
server_key: "fb417cf55c9f0442670c5ec64ed5842b"
server_key_iv: "5b86ba9c27ddafaf18118b9bce9ae2af"
server_key_version: "encryptionkey@0"
I20260927 03:58:35.062630 19221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:35.072700 19221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:35.074743 19221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:35.074911 19221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:35.075210 19221 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:58:35.075539 19221 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:58:35.075594 19221 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:35.075635 19221 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:58:35.075663 19221 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:35.081779 19221 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.65:41743
I20260927 03:58:35.081821 19349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.65:41743 every 8 connection(s)
I20260927 03:58:35.082161 19221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
I20260927 03:58:35.085415 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 19221
I20260927 03:58:35.085547 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-0/wal/instance
I20260927 03:58:35.085733 16101 external_mini_cluster.cc:1428] Setting key d16b56df76b52e684d2674ec64ffae01
Sep 27 03:58:35 dist-test-slave-glgz krb5kdc[19114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481515, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.65@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 03:58:35.092679 19201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:35.082849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:46619 (local address 127.15.185.126:44607)
0927 03:58:35.082918 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:35.082920 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:35.083137 (+ 217us) server_negotiation.cc:407] Connection header received
0927 03:58:35.083626 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:35.083627 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:35.083636 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:35.083697 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:35.084630 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:35.085123 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:35.086387 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:35.086637 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:35.086720 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:35.089324 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:35.089326 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:35.089326 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:35.089348 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:35.090963 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:35.091564 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:35.091565 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:35.091565 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:35.091629 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:35.091982 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:35.091982 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:35.091982 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:35.092102 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:35.092154 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:35.092530 (+ 376us) server_negotiation.cc:299] Negotiation successful
0927 03:58:35.092609 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 03:58:35.092952 19352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:35.082798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:44607 (local address 127.15.185.65:46619)
0927 03:58:35.082982 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:35.083028 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:35.083541 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:35.083753 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:35.083756 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:35.084339 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:35.084550 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:35.084556 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:35.085221 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:35.085222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:35.086313 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:35.086316 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:35.086726 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:35.086727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:35.086873 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:35.087467 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:35.087476 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:35.089211 (+ 1735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:35.091083 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:35.091092 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:35.091095 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:35.091464 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:35.091706 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:35.091707 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:35.091707 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:35.091900 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:35.092172 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:35.092178 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:35.092372 (+ 194us) client_negotiation.cc:769] Sending connection context
0927 03:58:35.092491 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 03:58:35.092678 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":105,"threads_started":1}
I20260927 03:58:35.093568 19350 heartbeater.cc:347] Connected to a master server at 127.15.185.126:44607
I20260927 03:58:35.093686 19350 heartbeater.cc:464] Registering TS with master...
I20260927 03:58:35.093900 19350 heartbeater.cc:511] Master 127.15.185.126:44607 requested a full tablet report, sending...
I20260927 03:58:35.094506 19159 ts_manager.cc:194] Registered new tserver with Master: d9f06c89fe8a46dfb67afc3ceaa940eb (127.15.185.65:41743)
WARNING: no policy specified for kudu/127.15.185.66@KRBTEST.COM; defaulting to no policy
I20260927 03:58:35.095500 19159 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.185.65@KRBTEST.COM'} at 127.15.185.65:46619
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab.
Entry for principal kudu/127.15.185.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab.
I20260927 03:58:35.114619 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.66:0
--local_ip_for_outbound_sockets=127.15.185.66
--webserver_interface=127.15.185.66
--webserver_port=0
--tserver_master_addrs=127.15.185.126:44607
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab
--principal=kudu/127.15.185.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:43089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:58:35.197476 19355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:35.197690 19355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:35.197714 19355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:35.198895 19355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:35.198941 19355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:35.198968 19355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.66
I20260927 03:58:35.200492 19355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:43089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab
--principal=kudu/127.15.185.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.185.66
--webserver_port=0
--tserver_master_addrs=127.15.185.126:44607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.66
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:35.200742 19355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:35.200977 19355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:35.206262 19361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:35.206257 19360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:35.206293 19363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:35.206518 19355 server_base.cc:1034] running on GCE node
I20260927 03:58:35.206714 19355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:35.206972 19355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:35.208149 19355 hybrid_clock.cc:648] HybridClock initialized: now 1790481515208140 us; error 54 us; skew 500 ppm
Sep 27 03:58:35 dist-test-slave-glgz krb5kdc[19114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481515, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:35.210932 19355 init.cc:377] Logged in from keytab as kudu/127.15.185.66@KRBTEST.COM (short username kudu)
I20260927 03:58:35.211426 19355 webserver.cc:460] Webserver started at http://127.15.185.66:35525/ using document root <none> and password file <none>
I20260927 03:58:35.211654 19355 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:35.211723 19355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:35.211874 19355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:35.213003 19355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-1/data/instance:
uuid: "f99b301daeb644c1bf586594c2631e0f"
format_stamp: "Formatted at 2026-09-27 03:58:35 on dist-test-slave-glgz"
server_key: "4ef0bdebd242af5d37a11689982514e8"
server_key_iv: "f28b1f02c38b664b28cc450bbac306b5"
server_key_version: "encryptionkey@0"
I20260927 03:58:35.213373 19355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-1/wal/instance:
uuid: "f99b301daeb644c1bf586594c2631e0f"
format_stamp: "Formatted at 2026-09-27 03:58:35 on dist-test-slave-glgz"
server_key: "4ef0bdebd242af5d37a11689982514e8"
server_key_iv: "f28b1f02c38b664b28cc450bbac306b5"
server_key_version: "encryptionkey@0"
I20260927 03:58:35.214761 19355 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:58:35.215777 19370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:35.216037 19355 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 03:58:35.216141 19355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-1/wal
uuid: "f99b301daeb644c1bf586594c2631e0f"
format_stamp: "Formatted at 2026-09-27 03:58:35 on dist-test-slave-glgz"
server_key: "4ef0bdebd242af5d37a11689982514e8"
server_key_iv: "f28b1f02c38b664b28cc450bbac306b5"
server_key_version: "encryptionkey@0"
I20260927 03:58:35.216225 19355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:35.248272 19355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:35.251545 19355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:35.251725 19355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:35.252033 19355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:58:35.252404 19355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:58:35.252467 19355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:35.252524 19355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:58:35.252554 19355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:35.258983 19355 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.66:44103
I20260927 03:58:35.259095 19483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.66:44103 every 8 connection(s)
I20260927 03:58:35.259457 19355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
I20260927 03:58:35.260095 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 19355
I20260927 03:58:35.260221 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-1/wal/instance
I20260927 03:58:35.260461 16101 external_mini_cluster.cc:1428] Setting key 64da97c1f86885771d8b3ca3b20f3ec2
Sep 27 03:58:35 dist-test-slave-glgz krb5kdc[19114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481515, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.66@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.185.67@KRBTEST.COM; defaulting to no policy
I20260927 03:58:35.270900 19201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:35.260408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:58749 (local address 127.15.185.126:44607)
0927 03:58:35.260453 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:35.260455 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:35.260787 (+ 332us) server_negotiation.cc:407] Connection header received
0927 03:58:35.261606 (+ 819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:35.261608 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:35.261623 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:35.261698 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:35.262868 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:35.263459 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:35.264499 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:35.264719 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:35.264810 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:35.267468 (+ 2658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:35.267470 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:35.267470 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:35.267479 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:35.269116 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:35.269730 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:35.269732 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:35.269733 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:35.269800 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:35.270105 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:35.270106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:35.270107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:35.270246 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:35.270330 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:35.270678 (+ 348us) server_negotiation.cc:299] Negotiation successful
0927 03:58:35.270778 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 03:58:35.271121 19486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:35.260423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:44607 (local address 127.15.185.66:58749)
0927 03:58:35.260642 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:35.260692 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:35.261470 (+ 778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:35.261772 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:35.261776 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:35.262563 (+ 787us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:35.262761 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:35.262766 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:35.263563 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:35.263564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:35.264390 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:35.264392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:35.264824 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:35.264825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:35.265001 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:35.265656 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:35.265666 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:35.267363 (+ 1697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:35.269226 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:35.269233 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:35.269236 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:35.269632 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:35.269898 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:35.269899 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:35.269899 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:35.270022 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:35.270319 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:35.270326 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:35.270515 (+ 189us) client_negotiation.cc:769] Sending connection context
0927 03:58:35.270653 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 03:58:35.270826 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":92,"threads_started":1}
I20260927 03:58:35.271677 19484 heartbeater.cc:347] Connected to a master server at 127.15.185.126:44607
I20260927 03:58:35.271790 19484 heartbeater.cc:464] Registering TS with master...
I20260927 03:58:35.271989 19484 heartbeater.cc:511] Master 127.15.185.126:44607 requested a full tablet report, sending...
I20260927 03:58:35.272431 19159 ts_manager.cc:194] Registered new tserver with Master: f99b301daeb644c1bf586594c2631e0f (127.15.185.66:44103)
I20260927 03:58:35.272811 19159 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.185.66@KRBTEST.COM'} at 127.15.185.66:58749
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab.
Entry for principal kudu/127.15.185.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab.
I20260927 03:58:35.290122 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.67:0
--local_ip_for_outbound_sockets=127.15.185.67
--webserver_interface=127.15.185.67
--webserver_port=0
--tserver_master_addrs=127.15.185.126:44607
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab
--principal=kudu/127.15.185.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:43089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:58:35.373136 19489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:35.373399 19489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:35.373445 19489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:35.374745 19489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:35.374818 19489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:35.374881 19489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.67
I20260927 03:58:35.376534 19489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:43089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab
--principal=kudu/127.15.185.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.185.67
--webserver_port=0
--tserver_master_addrs=127.15.185.126:44607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.67
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:35.376829 19489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:35.377102 19489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:35.382130 19497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:35.382148 19495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:35.382165 19494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:35.382506 19489 server_base.cc:1034] running on GCE node
I20260927 03:58:35.382728 19489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:35.382972 19489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:35.384131 19489 hybrid_clock.cc:648] HybridClock initialized: now 1790481515384121 us; error 27 us; skew 500 ppm
Sep 27 03:58:35 dist-test-slave-glgz krb5kdc[19114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481515, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:35.386590 19489 init.cc:377] Logged in from keytab as kudu/127.15.185.67@KRBTEST.COM (short username kudu)
I20260927 03:58:35.387171 19489 webserver.cc:460] Webserver started at http://127.15.185.67:33693/ using document root <none> and password file <none>
I20260927 03:58:35.387399 19489 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:35.387470 19489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:35.387624 19489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:35.388690 19489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-2/data/instance:
uuid: "2741a1bfd75141a68ff679b20453bb5c"
format_stamp: "Formatted at 2026-09-27 03:58:35 on dist-test-slave-glgz"
server_key: "14b7727479daf4ec3adfebbf94342046"
server_key_iv: "9e23f21aa17fdea75bb409247e4e7179"
server_key_version: "encryptionkey@0"
I20260927 03:58:35.389065 19489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-2/wal/instance:
uuid: "2741a1bfd75141a68ff679b20453bb5c"
format_stamp: "Formatted at 2026-09-27 03:58:35 on dist-test-slave-glgz"
server_key: "14b7727479daf4ec3adfebbf94342046"
server_key_iv: "9e23f21aa17fdea75bb409247e4e7179"
server_key_version: "encryptionkey@0"
I20260927 03:58:35.390483 19489 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:58:35.391361 19504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:35.391537 19489 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:58:35.391634 19489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-2/wal
uuid: "2741a1bfd75141a68ff679b20453bb5c"
format_stamp: "Formatted at 2026-09-27 03:58:35 on dist-test-slave-glgz"
server_key: "14b7727479daf4ec3adfebbf94342046"
server_key_iv: "9e23f21aa17fdea75bb409247e4e7179"
server_key_version: "encryptionkey@0"
I20260927 03:58:35.391716 19489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:35.406353 19489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:35.408792 19489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:35.408970 19489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:35.409204 19489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:58:35.409524 19489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:58:35.409577 19489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:35.409621 19489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:58:35.409647 19489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:35.415756 19489 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.67:41617
I20260927 03:58:35.415802 19617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.67:41617 every 8 connection(s)
I20260927 03:58:35.416124 19489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
Sep 27 03:58:35 dist-test-slave-glgz krb5kdc[19114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481515, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.67@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 03:58:35.425029 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 19489
I20260927 03:58:35.425133 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481498018407-16101-0/minicluster-data/ts-2/wal/instance
I20260927 03:58:35.425295 16101 external_mini_cluster.cc:1428] Setting key 3e9d585e53f0dec610f5c195be1e0a6c
I20260927 03:58:35.425652 19201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:35.416842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:43479 (local address 127.15.185.126:44607)
0927 03:58:35.416914 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:35.416916 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:35.417130 (+ 214us) server_negotiation.cc:407] Connection header received
0927 03:58:35.417728 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:35.417729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:35.417742 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:35.417792 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:35.418886 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:35.419511 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:35.420637 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:35.420843 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:35.420930 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:35.423020 (+ 2090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:35.423021 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:35.423021 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:35.423028 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:35.424266 (+ 1238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:35.424625 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:35.424625 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:35.424626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:35.424672 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:35.424903 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:35.424904 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:35.424904 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:35.425073 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:35.425149 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:35.425479 (+ 330us) server_negotiation.cc:299] Negotiation successful
0927 03:58:35.425558 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 03:58:35.425861 19620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:35.416811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:44607 (local address 127.15.185.67:43479)
0927 03:58:35.417000 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:35.417046 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:35.417594 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:35.417865 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:35.417869 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:35.418573 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:35.418776 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:35.418780 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:35.419615 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:35.419616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:35.420550 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:35.420553 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:35.420938 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:35.420939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:35.421068 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:35.421545 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:35.421553 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:35.422930 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:35.424336 (+ 1406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:35.424341 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:35.424342 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:35.424560 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:35.424734 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:35.424734 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:35.424735 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:35.424840 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:35.425148 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:35.425154 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:35.425299 (+ 145us) client_negotiation.cc:769] Sending connection context
0927 03:58:35.425454 (+ 155us) client_negotiation.cc:240] Negotiation successful
0927 03:58:35.425616 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":99,"threads_started":1}
I20260927 03:58:35.426471 19618 heartbeater.cc:347] Connected to a master server at 127.15.185.126:44607
I20260927 03:58:35.426584 19618 heartbeater.cc:464] Registering TS with master...
I20260927 03:58:35.426759 19618 heartbeater.cc:511] Master 127.15.185.126:44607 requested a full tablet report, sending...
I20260927 03:58:35.427160 19159 ts_manager.cc:194] Registered new tserver with Master: 2741a1bfd75141a68ff679b20453bb5c (127.15.185.67:41617)
I20260927 03:58:35.427559 19159 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.185.67@KRBTEST.COM'} at 127.15.185.67:43479
I20260927 03:58:35.437244 16101 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 03:58:35 dist-test-slave-glgz krb5kdc[19114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481515, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 27 03:58:35 dist-test-slave-glgz krb5kdc[19114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481515, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 03:58:35.459825 19201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:35.452477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54642 (local address 127.15.185.126:44607)
0927 03:58:35.452597 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:35.452600 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:35.452715 (+ 115us) server_negotiation.cc:407] Connection header received
0927 03:58:35.452827 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:35.452827 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:35.452839 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:35.452892 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:35.453510 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:35.454086 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:35.454884 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:35.455179 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:35.455255 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:35.457414 (+ 2159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:35.457416 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:35.457416 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:35.457424 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:35.458578 (+ 1154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:35.458973 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:35.458974 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:35.458974 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:35.459032 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:35.459260 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:35.459261 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:35.459261 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:35.459419 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:35.459527 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:35.459643 (+ 116us) server_negotiation.cc:299] Negotiation successful
0927 03:58:35.459722 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260927 03:58:35.460090 19159 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:54642
I20260927 03:58:35.460573 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 19221
I20260927 03:58:35.464738 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 19355
I20260927 03:58:35.469246 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 19489
I20260927 03:58:35.473814 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 19127
2026-09-27T03:58:35Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1866 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:58:35 dist-test-slave-glgz krb5kdc[19630](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:58:35 dist-test-slave-glgz krb5kdc[19630](info): set up 2 sockets
Sep 27 03:58:35 dist-test-slave-glgz krb5kdc[19630](info): commencing operation
krb5kdc: starting...
W20260927 03:58:36.515596 16101 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Sep 27 03:58:36 dist-test-slave-glgz krb5kdc[19630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:58:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:58:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.185.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab.
Entry for principal kudu/127.15.185.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab.
I20260927 03:58:36.642776 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:39447
--webserver_interface=127.15.185.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:36489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.185.126:39447
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:58:36.722699 19643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:36.722922 19643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:36.722945 19643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:36.724242 19643 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:58:36.724287 19643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:36.724303 19643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:36.724314 19643 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:58:36.724327 19643 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:58:36.725852 19643 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:36489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.185.126:39447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:39447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:36.726112 19643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:36.726359 19643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:36.731134 19651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:36.731169 19649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:36.731174 19643 server_base.cc:1034] running on GCE node
W20260927 03:58:36.731146 19648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:36.731571 19643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:36.731855 19643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:36.733024 19643 hybrid_clock.cc:648] HybridClock initialized: now 1790481516733003 us; error 39 us; skew 500 ppm
Sep 27 03:58:36 dist-test-slave-glgz krb5kdc[19630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481516, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:36.735479 19643 init.cc:377] Logged in from keytab as kudu/127.15.185.126@KRBTEST.COM (short username kudu)
I20260927 03:58:36.735857 19643 webserver.cc:460] Webserver started at http://127.15.185.126:38523/ using document root <none> and password file <none>
I20260927 03:58:36.736050 19643 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:36.736090 19643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:36.736179 19643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:36.737294 19643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/minicluster-data/master-0/data/instance:
uuid: "a2e159e9ea0e49f89fea91bd50d06ebd"
format_stamp: "Formatted at 2026-09-27 03:58:36 on dist-test-slave-glgz"
server_key: "6bcf017203cd6b4db00df8e861079f35"
server_key_iv: "54a144efef2ca9e06404e12c4f6c70e7"
server_key_version: "encryptionkey@0"
I20260927 03:58:36.737603 19643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/minicluster-data/master-0/wal/instance:
uuid: "a2e159e9ea0e49f89fea91bd50d06ebd"
format_stamp: "Formatted at 2026-09-27 03:58:36 on dist-test-slave-glgz"
server_key: "6bcf017203cd6b4db00df8e861079f35"
server_key_iv: "54a144efef2ca9e06404e12c4f6c70e7"
server_key_version: "encryptionkey@0"
I20260927 03:58:36.738852 19643 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:58:36.739669 19658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:36.739921 19643 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 03:58:36.740001 19643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/minicluster-data/master-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/minicluster-data/master-0/wal
uuid: "a2e159e9ea0e49f89fea91bd50d06ebd"
format_stamp: "Formatted at 2026-09-27 03:58:36 on dist-test-slave-glgz"
server_key: "6bcf017203cd6b4db00df8e861079f35"
server_key_iv: "54a144efef2ca9e06404e12c4f6c70e7"
server_key_version: "encryptionkey@0"
I20260927 03:58:36.740088 19643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:36.767797 19643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:36.771239 19643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:36.771394 19643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:36.776568 19643 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.126:39447
I20260927 03:58:36.776588 19710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.126:39447 every 8 connection(s)
I20260927 03:58:36.777120 19643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
I20260927 03:58:36.777616 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 19643
I20260927 03:58:36.777853 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/minicluster-data/master-0/wal/instance
I20260927 03:58:36.778057 19711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:36.778090 16101 external_mini_cluster.cc:1428] Setting key 41e52b5829e741679a27d2c24b2db51f
I20260927 03:58:36.782716 19711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2e159e9ea0e49f89fea91bd50d06ebd: Bootstrap starting.
I20260927 03:58:36.783587 19711 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2e159e9ea0e49f89fea91bd50d06ebd: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:36.783921 19711 log.cc:824] T 00000000000000000000000000000000 P a2e159e9ea0e49f89fea91bd50d06ebd: Log is configured to *not* fsync() on all Append() calls
Sep 27 03:58:36 dist-test-slave-glgz krb5kdc[19630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 03:58:36.784734 19711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2e159e9ea0e49f89fea91bd50d06ebd: No bootstrap required, opened a new log
I20260927 03:58:36.787410 19711 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2e159e9ea0e49f89fea91bd50d06ebd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e159e9ea0e49f89fea91bd50d06ebd" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 39447 } }
I20260927 03:58:36.787606 19711 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2e159e9ea0e49f89fea91bd50d06ebd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:36.787648 19711 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2e159e9ea0e49f89fea91bd50d06ebd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e159e9ea0e49f89fea91bd50d06ebd, State: Initialized, Role: FOLLOWER
I20260927 03:58:36.787817 19711 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2e159e9ea0e49f89fea91bd50d06ebd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e159e9ea0e49f89fea91bd50d06ebd" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 39447 } }
I20260927 03:58:36.787930 19711 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2e159e9ea0e49f89fea91bd50d06ebd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:58:36.787977 19711 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2e159e9ea0e49f89fea91bd50d06ebd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:58:36.788111 19711 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2e159e9ea0e49f89fea91bd50d06ebd [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:36.788998 19711 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2e159e9ea0e49f89fea91bd50d06ebd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e159e9ea0e49f89fea91bd50d06ebd" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 39447 } }
I20260927 03:58:36.789134 19711 leader_election.cc:302] T 00000000000000000000000000000000 P a2e159e9ea0e49f89fea91bd50d06ebd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2e159e9ea0e49f89fea91bd50d06ebd; no voters:
I20260927 03:58:36.789280 19714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:36.778600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50408 (local address 127.15.185.126:39447)
0927 03:58:36.778807 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:36.778814 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:36.778839 (+ 25us) server_negotiation.cc:407] Connection header received
0927 03:58:36.779461 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:36.779471 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:36.779523 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:36.779677 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:36.780305 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:36.781271 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:36.782047 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:36.782453 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:36.782582 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:36.785220 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:36.785224 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:36.785225 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:36.785238 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:36.787335 (+ 2097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:36.787713 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:36.787716 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:36.787716 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:36.787829 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:36.788106 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:36.788108 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:36.788108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:36.788478 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:36.788628 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:36.788813 (+ 185us) server_negotiation.cc:299] Negotiation successful
0927 03:58:36.788995 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":103,"threads_started":1}
I20260927 03:58:36.789346 19711 leader_election.cc:288] T 00000000000000000000000000000000 P a2e159e9ea0e49f89fea91bd50d06ebd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:58:36.789529 19717 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2e159e9ea0e49f89fea91bd50d06ebd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:58:36.789616 19711 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2e159e9ea0e49f89fea91bd50d06ebd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:58:36.789659 19717 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2e159e9ea0e49f89fea91bd50d06ebd [term 1 LEADER]: Becoming Leader. State: Replica: a2e159e9ea0e49f89fea91bd50d06ebd, State: Running, Role: LEADER
I20260927 03:58:36.789748 19717 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2e159e9ea0e49f89fea91bd50d06ebd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e159e9ea0e49f89fea91bd50d06ebd" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 39447 } }
I20260927 03:58:36.790126 19716 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2e159e9ea0e49f89fea91bd50d06ebd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2e159e9ea0e49f89fea91bd50d06ebd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e159e9ea0e49f89fea91bd50d06ebd" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 39447 } } }
I20260927 03:58:36.790268 19716 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2e159e9ea0e49f89fea91bd50d06ebd [sys.catalog]: This master's current role is: LEADER
I20260927 03:58:36.790498 19718 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2e159e9ea0e49f89fea91bd50d06ebd [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2e159e9ea0e49f89fea91bd50d06ebd. Latest consensus state: current_term: 1 leader_uuid: "a2e159e9ea0e49f89fea91bd50d06ebd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e159e9ea0e49f89fea91bd50d06ebd" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 39447 } } }
I20260927 03:58:36.790544 19725 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:58:36.790627 19718 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2e159e9ea0e49f89fea91bd50d06ebd [sys.catalog]: This master's current role is: LEADER
I20260927 03:58:36.791147 19725 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:58:36.792879 19725 catalog_manager.cc:1347] Generated new cluster ID: 029a0fff4a514e00a782eb1c67bb8d7c
I20260927 03:58:36.792937 19725 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:58:36.801301 19725 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:58:36.801858 19725 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:58:36.805225 19725 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2e159e9ea0e49f89fea91bd50d06ebd: Generated new TSK 0
I20260927 03:58:36.805408 19725 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.185.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab.
Entry for principal kudu/127.15.185.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab.
I20260927 03:58:36.838176 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.65:0
--local_ip_for_outbound_sockets=127.15.185.65
--webserver_interface=127.15.185.65
--webserver_port=0
--tserver_master_addrs=127.15.185.126:39447
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab
--principal=kudu/127.15.185.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:36489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:58:36.939983 19737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:36.940248 19737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:36.940289 19737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:36.941493 19737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:36.941572 19737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:36.941620 19737 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.65
I20260927 03:58:36.943451 19737 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:36489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab
--principal=kudu/127.15.185.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.185.65
--webserver_port=0
--tserver_master_addrs=127.15.185.126:39447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19737
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.65
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:36.943744 19737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:36.943989 19737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:36.949131 19745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:36.949127 19742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:36.949129 19743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:36.949331 19737 server_base.cc:1034] running on GCE node
I20260927 03:58:36.949743 19737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:36.950030 19737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:36.951189 19737 hybrid_clock.cc:648] HybridClock initialized: now 1790481516951177 us; error 36 us; skew 500 ppm
Sep 27 03:58:36 dist-test-slave-glgz krb5kdc[19630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481516, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:36.953905 19737 init.cc:377] Logged in from keytab as kudu/127.15.185.65@KRBTEST.COM (short username kudu)
I20260927 03:58:36.954361 19737 webserver.cc:460] Webserver started at http://127.15.185.65:36691/ using document root <none> and password file <none>
I20260927 03:58:36.954618 19737 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:36.954689 19737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:36.954829 19737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:36.956020 19737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/minicluster-data/ts-0/data/instance:
uuid: "9ecb0174133e43f485efc6285233d440"
format_stamp: "Formatted at 2026-09-27 03:58:36 on dist-test-slave-glgz"
server_key: "6209e491a07cfd5dd7d14c5402c7a67d"
server_key_iv: "9aa9b748f76a51b5df580e6a7d69d2e7"
server_key_version: "encryptionkey@0"
I20260927 03:58:36.956426 19737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/minicluster-data/ts-0/wal/instance:
uuid: "9ecb0174133e43f485efc6285233d440"
format_stamp: "Formatted at 2026-09-27 03:58:36 on dist-test-slave-glgz"
server_key: "6209e491a07cfd5dd7d14c5402c7a67d"
server_key_iv: "9aa9b748f76a51b5df580e6a7d69d2e7"
server_key_version: "encryptionkey@0"
I20260927 03:58:36.957806 19737 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:58:36.958557 19752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:36.958719 19737 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:58:36.958823 19737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/minicluster-data/ts-0/wal
uuid: "9ecb0174133e43f485efc6285233d440"
format_stamp: "Formatted at 2026-09-27 03:58:36 on dist-test-slave-glgz"
server_key: "6209e491a07cfd5dd7d14c5402c7a67d"
server_key_iv: "9aa9b748f76a51b5df580e6a7d69d2e7"
server_key_version: "encryptionkey@0"
I20260927 03:58:36.958896 19737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:36.980154 19737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:36.982559 19737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:36.982720 19737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:36.982965 19737 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:58:36.983306 19737 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:58:36.983359 19737 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:36.983400 19737 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:58:36.983424 19737 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:36.989122 19737 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.65:45719
I20260927 03:58:36.989203 19865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.65:45719 every 8 connection(s)
I20260927 03:58:36.989532 19737 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
I20260927 03:58:36.993857 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 19737
I20260927 03:58:36.993999 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/minicluster-data/ts-0/wal/instance
I20260927 03:58:36.994220 16101 external_mini_cluster.cc:1428] Setting key 4823cebb8a56d777fdfb667e28ed8c57
Sep 27 03:58:36 dist-test-slave-glgz krb5kdc[19630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481516, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.65@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 03:58:36.999588 19714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:36.990435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:54263 (local address 127.15.185.126:39447)
0927 03:58:36.990489 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:36.990491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:36.990723 (+ 232us) server_negotiation.cc:407] Connection header received
0927 03:58:36.991767 (+ 1044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:36.991768 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:36.991779 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:36.991833 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:36.992428 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:36.992943 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:36.994016 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:36.994331 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:36.994416 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:36.996668 (+ 2252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:36.996669 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:36.996670 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:36.996678 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:36.998171 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:36.998590 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:36.998591 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:36.998592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:36.998650 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:36.998914 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:36.998915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:36.998915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:36.999031 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:36.999126 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:36.999403 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 03:58:36.999488 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 03:58:36.999806 19868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:36.990344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:39447 (local address 127.15.185.65:54263)
0927 03:58:36.990547 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:36.990584 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:36.991675 (+ 1091us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:36.991890 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:36.991894 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:36.992158 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:36.992355 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:36.992359 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:36.993038 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:36.993039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:36.993903 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:36.993905 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:36.994430 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:36.994432 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:36.994579 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:36.995187 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:36.995201 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:36.996569 (+ 1368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:36.998263 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:36.998269 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:36.998272 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:36.998499 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:36.998720 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:36.998721 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:36.998721 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:36.998839 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:36.999117 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:36.999123 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:36.999268 (+ 145us) client_negotiation.cc:769] Sending connection context
0927 03:58:36.999386 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 03:58:36.999553 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":81,"threads_started":1}
I20260927 03:58:37.000450 19866 heartbeater.cc:347] Connected to a master server at 127.15.185.126:39447
I20260927 03:58:37.000535 19866 heartbeater.cc:464] Registering TS with master...
I20260927 03:58:37.000720 19866 heartbeater.cc:511] Master 127.15.185.126:39447 requested a full tablet report, sending...
I20260927 03:58:37.001308 19675 ts_manager.cc:194] Registered new tserver with Master: 9ecb0174133e43f485efc6285233d440 (127.15.185.65:45719)
WARNING: no policy specified for kudu/127.15.185.66@KRBTEST.COM; defaulting to no policy
I20260927 03:58:37.002297 19675 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.185.65@KRBTEST.COM'} at 127.15.185.65:54263
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab.
Entry for principal kudu/127.15.185.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab.
I20260927 03:58:37.021492 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.66:0
--local_ip_for_outbound_sockets=127.15.185.66
--webserver_interface=127.15.185.66
--webserver_port=0
--tserver_master_addrs=127.15.185.126:39447
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab
--principal=kudu/127.15.185.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:36489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:58:37.103215 19871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:37.103432 19871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:37.103456 19871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:37.104669 19871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:37.104713 19871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:37.104739 19871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.66
I20260927 03:58:37.106258 19871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:36489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab
--principal=kudu/127.15.185.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.185.66
--webserver_port=0
--tserver_master_addrs=127.15.185.126:39447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.66
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:37.106516 19871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:37.106767 19871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:37.112040 19879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:37.112040 19877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:37.112040 19876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:37.112437 19871 server_base.cc:1034] running on GCE node
I20260927 03:58:37.112612 19871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:37.112812 19871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:37.113965 19871 hybrid_clock.cc:648] HybridClock initialized: now 1790481517113942 us; error 35 us; skew 500 ppm
Sep 27 03:58:37 dist-test-slave-glgz krb5kdc[19630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481517, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:37.116536 19871 init.cc:377] Logged in from keytab as kudu/127.15.185.66@KRBTEST.COM (short username kudu)
I20260927 03:58:37.117004 19871 webserver.cc:460] Webserver started at http://127.15.185.66:35273/ using document root <none> and password file <none>
I20260927 03:58:37.117241 19871 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:37.117311 19871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:37.117465 19871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:37.118589 19871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/minicluster-data/ts-1/data/instance:
uuid: "283f23ce73bb4ca7936066a993b733de"
format_stamp: "Formatted at 2026-09-27 03:58:37 on dist-test-slave-glgz"
server_key: "156e59d808a81b0b5094dc97e661bddb"
server_key_iv: "7e92cc06534078d510fc6c8e11df8a7d"
server_key_version: "encryptionkey@0"
I20260927 03:58:37.118961 19871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/minicluster-data/ts-1/wal/instance:
uuid: "283f23ce73bb4ca7936066a993b733de"
format_stamp: "Formatted at 2026-09-27 03:58:37 on dist-test-slave-glgz"
server_key: "156e59d808a81b0b5094dc97e661bddb"
server_key_iv: "7e92cc06534078d510fc6c8e11df8a7d"
server_key_version: "encryptionkey@0"
I20260927 03:58:37.120424 19871 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:58:37.121227 19886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:37.121407 19871 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:58:37.121506 19871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/minicluster-data/ts-1/wal
uuid: "283f23ce73bb4ca7936066a993b733de"
format_stamp: "Formatted at 2026-09-27 03:58:37 on dist-test-slave-glgz"
server_key: "156e59d808a81b0b5094dc97e661bddb"
server_key_iv: "7e92cc06534078d510fc6c8e11df8a7d"
server_key_version: "encryptionkey@0"
I20260927 03:58:37.121603 19871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:37.138650 19871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:37.140856 19871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:37.141037 19871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:37.141281 19871 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:58:37.141620 19871 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:58:37.141675 19871 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:37.141718 19871 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:58:37.141746 19871 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:37.147825 19871 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.66:44439
I20260927 03:58:37.147998 19999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.66:44439 every 8 connection(s)
I20260927 03:58:37.148260 19871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
Sep 27 03:58:37 dist-test-slave-glgz krb5kdc[19630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481517, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.66@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 03:58:37.156962 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 19871
I20260927 03:58:37.157087 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/minicluster-data/ts-1/wal/instance
I20260927 03:58:37.157323 16101 external_mini_cluster.cc:1428] Setting key 3f4473f2228231217abef6bdcc4b97f1
I20260927 03:58:37.158922 19714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:37.149112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:55083 (local address 127.15.185.126:39447)
0927 03:58:37.149222 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:37.149225 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:37.149441 (+ 216us) server_negotiation.cc:407] Connection header received
0927 03:58:37.150483 (+ 1042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:37.150484 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:37.150497 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:37.150556 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:37.151351 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:37.151921 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:37.153058 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:37.153292 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:37.153383 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:37.155703 (+ 2320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:37.155706 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:37.155706 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:37.155715 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:37.157301 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:37.157849 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:37.157850 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:37.157851 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:37.157922 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:37.158253 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:37.158254 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:37.158254 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:37.158370 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:37.158427 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:37.158781 (+ 354us) server_negotiation.cc:299] Negotiation successful
0927 03:58:37.158856 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 03:58:37.159261 20002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:37.149081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:39447 (local address 127.15.185.66:55083)
0927 03:58:37.149279 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:37.149321 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:37.150361 (+ 1040us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:37.150654 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:37.150658 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:37.150960 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:37.151247 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:37.151252 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:37.152033 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:37.152034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:37.152944 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:37.152946 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:37.153389 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:37.153391 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:37.153537 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:37.154066 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:37.154075 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:37.155598 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:37.157404 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:37.157411 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:37.157413 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:37.157777 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:37.158010 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:37.158011 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:37.158012 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:37.158180 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:37.158455 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:37.158460 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:37.158639 (+ 179us) client_negotiation.cc:769] Sending connection context
0927 03:58:37.158762 (+ 123us) client_negotiation.cc:240] Negotiation successful
0927 03:58:37.158947 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":80,"threads_started":1}
I20260927 03:58:37.159768 20000 heartbeater.cc:347] Connected to a master server at 127.15.185.126:39447
I20260927 03:58:37.159895 20000 heartbeater.cc:464] Registering TS with master...
I20260927 03:58:37.160080 20000 heartbeater.cc:511] Master 127.15.185.126:39447 requested a full tablet report, sending...
I20260927 03:58:37.160470 19675 ts_manager.cc:194] Registered new tserver with Master: 283f23ce73bb4ca7936066a993b733de (127.15.185.66:44439)
I20260927 03:58:37.160861 19675 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.185.66@KRBTEST.COM'} at 127.15.185.66:55083
WARNING: no policy specified for kudu/127.15.185.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab.
Entry for principal kudu/127.15.185.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab.
I20260927 03:58:37.183128 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.67:0
--local_ip_for_outbound_sockets=127.15.185.67
--webserver_interface=127.15.185.67
--webserver_port=0
--tserver_master_addrs=127.15.185.126:39447
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab
--principal=kudu/127.15.185.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:36489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:58:37.266589 20005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:37.266804 20005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:37.266829 20005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:37.268069 20005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:37.268115 20005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:37.268143 20005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.67
I20260927 03:58:37.269565 20005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:36489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab
--principal=kudu/127.15.185.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.185.67
--webserver_port=0
--tserver_master_addrs=127.15.185.126:39447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.67
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:37.269752 20005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:37.269934 20005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:37.274806 20013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:37.274804 20011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:37.274878 20010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:37.275095 20005 server_base.cc:1034] running on GCE node
I20260927 03:58:37.275310 20005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:37.275512 20005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:37.276655 20005 hybrid_clock.cc:648] HybridClock initialized: now 1790481517276642 us; error 27 us; skew 500 ppm
Sep 27 03:58:37 dist-test-slave-glgz krb5kdc[19630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481517, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:37.279246 20005 init.cc:377] Logged in from keytab as kudu/127.15.185.67@KRBTEST.COM (short username kudu)
I20260927 03:58:37.279626 20005 webserver.cc:460] Webserver started at http://127.15.185.67:43413/ using document root <none> and password file <none>
I20260927 03:58:37.279856 20005 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:37.279904 20005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:37.280045 20005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:37.281109 20005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/minicluster-data/ts-2/data/instance:
uuid: "7568db4b7d254695bd91bb0b1d905a06"
format_stamp: "Formatted at 2026-09-27 03:58:37 on dist-test-slave-glgz"
server_key: "6549879248708d574dfe990c085c9085"
server_key_iv: "d7e02a04a8c867c1e7762ef6429ce049"
server_key_version: "encryptionkey@0"
I20260927 03:58:37.281471 20005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/minicluster-data/ts-2/wal/instance:
uuid: "7568db4b7d254695bd91bb0b1d905a06"
format_stamp: "Formatted at 2026-09-27 03:58:37 on dist-test-slave-glgz"
server_key: "6549879248708d574dfe990c085c9085"
server_key_iv: "d7e02a04a8c867c1e7762ef6429ce049"
server_key_version: "encryptionkey@0"
I20260927 03:58:37.282781 20005 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:58:37.283582 20020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:37.283751 20005 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:37.283847 20005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/minicluster-data/ts-2/wal
uuid: "7568db4b7d254695bd91bb0b1d905a06"
format_stamp: "Formatted at 2026-09-27 03:58:37 on dist-test-slave-glgz"
server_key: "6549879248708d574dfe990c085c9085"
server_key_iv: "d7e02a04a8c867c1e7762ef6429ce049"
server_key_version: "encryptionkey@0"
I20260927 03:58:37.283929 20005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:37.300166 20005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:37.302619 20005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:37.302785 20005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:37.303030 20005 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:58:37.303433 20005 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:58:37.303486 20005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:37.303529 20005 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:58:37.303556 20005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:37.309865 20005 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.67:44767
I20260927 03:58:37.309903 20133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.67:44767 every 8 connection(s)
I20260927 03:58:37.310256 20005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
Sep 27 03:58:37 dist-test-slave-glgz krb5kdc[19630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481517, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.67@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 03:58:37.317934 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 20005
I20260927 03:58:37.318037 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481498018407-16101-0/minicluster-data/ts-2/wal/instance
I20260927 03:58:37.318231 16101 external_mini_cluster.cc:1428] Setting key 4f63adb8625aa77d67d4b3262276baaf
I20260927 03:58:37.320638 19714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:37.311092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:35741 (local address 127.15.185.126:39447)
0927 03:58:37.311135 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:37.311137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:37.311388 (+ 251us) server_negotiation.cc:407] Connection header received
0927 03:58:37.312429 (+ 1041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:37.312430 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:37.312445 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:37.312497 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:37.313134 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:37.313654 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:37.314976 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:37.315276 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:37.315364 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:37.317500 (+ 2136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:37.317502 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:37.317502 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:37.317511 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:37.319264 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:37.319668 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:37.319669 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:37.319670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:37.319726 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:37.319979 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:37.319980 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:37.319981 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:37.320104 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:37.320181 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:37.320466 (+ 285us) server_negotiation.cc:299] Negotiation successful
0927 03:58:37.320551 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 03:58:37.320832 20136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:37.311036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:39447 (local address 127.15.185.67:35741)
0927 03:58:37.311241 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:37.311278 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:37.312307 (+ 1029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:37.312575 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:37.312579 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:37.312839 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:37.313041 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:37.313045 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:37.313772 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:37.313773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:37.314847 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:37.314850 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:37.315374 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:37.315375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:37.315504 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:37.316049 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:37.316058 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:37.317394 (+ 1336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:37.319338 (+ 1944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:37.319344 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:37.319347 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:37.319578 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:37.319800 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:37.319801 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:37.319802 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:37.319914 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:37.320170 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:37.320177 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:37.320308 (+ 131us) client_negotiation.cc:769] Sending connection context
0927 03:58:37.320411 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 03:58:37.320576 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":81,"threads_started":1}
I20260927 03:58:37.321271 20134 heartbeater.cc:347] Connected to a master server at 127.15.185.126:39447
I20260927 03:58:37.321378 20134 heartbeater.cc:464] Registering TS with master...
I20260927 03:58:37.321533 20134 heartbeater.cc:511] Master 127.15.185.126:39447 requested a full tablet report, sending...
I20260927 03:58:37.321851 19675 ts_manager.cc:194] Registered new tserver with Master: 7568db4b7d254695bd91bb0b1d905a06 (127.15.185.67:44767)
I20260927 03:58:37.322183 19675 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.185.67@KRBTEST.COM'} at 127.15.185.67:35741
I20260927 03:58:37.330272 16101 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 03:58:37 dist-test-slave-glgz krb5kdc[19630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 03:58:37 dist-test-slave-glgz krb5kdc[19630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.185.65@KRBTEST.COM
I20260927 03:58:37.352811 20141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:37.345373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50704 (local address 127.15.185.65:45719)
0927 03:58:37.345538 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:37.345543 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:37.345554 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:58:37.345580 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:37.345582 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:37.345596 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:37.345663 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:37.346262 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:37.347028 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:37.347793 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:37.348125 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:37.348183 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:37.350354 (+ 2171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:37.350361 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:37.350361 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:37.350367 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:37.351835 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:37.352163 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:37.352166 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:37.352166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:37.352229 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:37.352390 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:37.352391 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:37.352391 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:37.352510 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:37.352618 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:37.352678 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 03:58:37.352734 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":88,"threads_started":1}
Sep 27 03:58:37 dist-test-slave-glgz krb5kdc[19630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 03:58:37.360987 19714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:37.354141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50414 (local address 127.15.185.126:39447)
0927 03:58:37.354193 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:37.354195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:37.354327 (+ 132us) server_negotiation.cc:407] Connection header received
0927 03:58:37.354365 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:37.354366 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:37.354375 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:37.354424 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:37.354893 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:37.355470 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:37.356180 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:37.356372 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:37.356428 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:37.358529 (+ 2101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:37.358531 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:37.358531 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:37.358539 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:37.359817 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:37.360171 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:37.360172 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:37.360172 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:37.360229 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:37.360416 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:37.360417 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:37.360417 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:37.360571 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:37.360685 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:37.360779 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 03:58:37.360876 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260927 03:58:37.361274 19675 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50414
I20260927 03:58:37.361632 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 19737
I20260927 03:58:37.366698 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 19871
I20260927 03:58:37.370831 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 20005
I20260927 03:58:37.375005 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 19643
2026-09-27T03:58:37Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1901 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:58:37 dist-test-slave-glgz krb5kdc[20146](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:58:37 dist-test-slave-glgz krb5kdc[20146](info): set up 2 sockets
Sep 27 03:58:37 dist-test-slave-glgz krb5kdc[20146](info): commencing operation
krb5kdc: starting...
W20260927 03:58:38.407284 16101 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Sep 27 03:58:38 dist-test-slave-glgz krb5kdc[20146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:58:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:58:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.185.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab.
Entry for principal kudu/127.15.185.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab.
I20260927 03:58:38.525738 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:45225
--webserver_interface=127.15.185.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:40343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.185.126:45225
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:58:38.605587 20159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:38.605782 20159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:38.605803 20159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:38.606873 20159 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:58:38.606904 20159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:38.606917 20159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:38.606930 20159 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:58:38.606941 20159 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:58:38.608371 20159 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:40343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.185.126:45225
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:45225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:38.608564 20159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:38.608724 20159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:38.613502 20165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:38.613476 20167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:38.613622 20164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:38.613749 20159 server_base.cc:1034] running on GCE node
I20260927 03:58:38.613977 20159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:38.614264 20159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:38.615432 20159 hybrid_clock.cc:648] HybridClock initialized: now 1790481518615415 us; error 29 us; skew 500 ppm
Sep 27 03:58:38 dist-test-slave-glgz krb5kdc[20146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481518, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:38.621692 20159 init.cc:377] Logged in from keytab as kudu/127.15.185.126@KRBTEST.COM (short username kudu)
I20260927 03:58:38.622092 20159 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:38.622160 20159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:38.622294 20159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:38.623421 20159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/master-0/data/instance:
uuid: "d19ec021c9694877a39c8d0d8e5124fd"
format_stamp: "Formatted at 2026-09-27 03:58:38 on dist-test-slave-glgz"
server_key: "2d5fc789bf2727ce8def1a304ad00a93"
server_key_iv: "43a7d4702871916b785f9d239eeffcc2"
server_key_version: "encryptionkey@0"
I20260927 03:58:38.623785 20159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/master-0/wal/instance:
uuid: "d19ec021c9694877a39c8d0d8e5124fd"
format_stamp: "Formatted at 2026-09-27 03:58:38 on dist-test-slave-glgz"
server_key: "2d5fc789bf2727ce8def1a304ad00a93"
server_key_iv: "43a7d4702871916b785f9d239eeffcc2"
server_key_version: "encryptionkey@0"
I20260927 03:58:38.625101 20159 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:58:38.625870 20173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:38.626083 20159 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:58:38.626180 20159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/master-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/master-0/wal
uuid: "d19ec021c9694877a39c8d0d8e5124fd"
format_stamp: "Formatted at 2026-09-27 03:58:38 on dist-test-slave-glgz"
server_key: "2d5fc789bf2727ce8def1a304ad00a93"
server_key_iv: "43a7d4702871916b785f9d239eeffcc2"
server_key_version: "encryptionkey@0"
I20260927 03:58:38.626256 20159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:38.644052 20159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:38.646440 20159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:38.646620 20159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:38.650655 20159 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.126:45225
I20260927 03:58:38.650684 20225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.126:45225 every 8 connection(s)
I20260927 03:58:38.650972 20159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
I20260927 03:58:38.651971 20226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:38.654420 20226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d19ec021c9694877a39c8d0d8e5124fd: Bootstrap starting.
I20260927 03:58:38.655031 20226 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d19ec021c9694877a39c8d0d8e5124fd: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:38.655368 20226 log.cc:824] T 00000000000000000000000000000000 P d19ec021c9694877a39c8d0d8e5124fd: Log is configured to *not* fsync() on all Append() calls
I20260927 03:58:38.656095 20226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d19ec021c9694877a39c8d0d8e5124fd: No bootstrap required, opened a new log
I20260927 03:58:38.657652 20226 raft_consensus.cc:348] T 00000000000000000000000000000000 P d19ec021c9694877a39c8d0d8e5124fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d19ec021c9694877a39c8d0d8e5124fd" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45225 } }
I20260927 03:58:38.657792 20226 raft_consensus.cc:374] T 00000000000000000000000000000000 P d19ec021c9694877a39c8d0d8e5124fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:38.657816 20226 raft_consensus.cc:729] T 00000000000000000000000000000000 P d19ec021c9694877a39c8d0d8e5124fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d19ec021c9694877a39c8d0d8e5124fd, State: Initialized, Role: FOLLOWER
I20260927 03:58:38.658010 20226 consensus_queue.cc:260] T 00000000000000000000000000000000 P d19ec021c9694877a39c8d0d8e5124fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d19ec021c9694877a39c8d0d8e5124fd" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45225 } }
I20260927 03:58:38.658070 20226 raft_consensus.cc:388] T 00000000000000000000000000000000 P d19ec021c9694877a39c8d0d8e5124fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:58:38.658125 20226 raft_consensus.cc:482] T 00000000000000000000000000000000 P d19ec021c9694877a39c8d0d8e5124fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:58:38.658174 20226 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d19ec021c9694877a39c8d0d8e5124fd [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:38.659026 20226 raft_consensus.cc:504] T 00000000000000000000000000000000 P d19ec021c9694877a39c8d0d8e5124fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d19ec021c9694877a39c8d0d8e5124fd" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45225 } }
I20260927 03:58:38.659199 20226 leader_election.cc:302] T 00000000000000000000000000000000 P d19ec021c9694877a39c8d0d8e5124fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d19ec021c9694877a39c8d0d8e5124fd; no voters:
I20260927 03:58:38.659389 20226 leader_election.cc:288] T 00000000000000000000000000000000 P d19ec021c9694877a39c8d0d8e5124fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:58:38.659454 20229 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d19ec021c9694877a39c8d0d8e5124fd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:58:38.659636 20229 raft_consensus.cc:686] T 00000000000000000000000000000000 P d19ec021c9694877a39c8d0d8e5124fd [term 1 LEADER]: Becoming Leader. State: Replica: d19ec021c9694877a39c8d0d8e5124fd, State: Running, Role: LEADER
I20260927 03:58:38.659653 20226 sys_catalog.cc:564] T 00000000000000000000000000000000 P d19ec021c9694877a39c8d0d8e5124fd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:58:38.659726 20229 consensus_queue.cc:237] T 00000000000000000000000000000000 P d19ec021c9694877a39c8d0d8e5124fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d19ec021c9694877a39c8d0d8e5124fd" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45225 } }
I20260927 03:58:38.660080 20230 sys_catalog.cc:455] T 00000000000000000000000000000000 P d19ec021c9694877a39c8d0d8e5124fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d19ec021c9694877a39c8d0d8e5124fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d19ec021c9694877a39c8d0d8e5124fd" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45225 } } }
I20260927 03:58:38.660084 20231 sys_catalog.cc:455] T 00000000000000000000000000000000 P d19ec021c9694877a39c8d0d8e5124fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader d19ec021c9694877a39c8d0d8e5124fd. Latest consensus state: current_term: 1 leader_uuid: "d19ec021c9694877a39c8d0d8e5124fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d19ec021c9694877a39c8d0d8e5124fd" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45225 } } }
I20260927 03:58:38.660204 20230 sys_catalog.cc:458] T 00000000000000000000000000000000 P d19ec021c9694877a39c8d0d8e5124fd [sys.catalog]: This master's current role is: LEADER
I20260927 03:58:38.660208 20231 sys_catalog.cc:458] T 00000000000000000000000000000000 P d19ec021c9694877a39c8d0d8e5124fd [sys.catalog]: This master's current role is: LEADER
I20260927 03:58:38.660457 20238 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:58:38.660588 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 20159
I20260927 03:58:38.660722 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/master-0/wal/instance
I20260927 03:58:38.660939 16101 external_mini_cluster.cc:1428] Setting key 0775eda3950d0de4a7c5301a60fa20b9
I20260927 03:58:38.660995 20238 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:58:38.662932 20238 catalog_manager.cc:1347] Generated new cluster ID: ede6cf42bd4149f9adfb4b3a5145f9d1
I20260927 03:58:38.663014 20238 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 03:58:38 dist-test-slave-glgz krb5kdc[20146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 03:58:38.671329 20248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:38.661393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40214 (local address 127.15.185.126:45225)
0927 03:58:38.661719 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:38.661728 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:38.661753 (+ 25us) server_negotiation.cc:407] Connection header received
0927 03:58:38.661852 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:38.661857 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:38.661900 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:38.662010 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:38.662784 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:38.664041 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:38.664864 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:38.665323 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:38.665457 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:38.667931 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:38.667935 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:38.667936 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:38.667949 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:38.669694 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:38.670078 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:38.670082 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:38.670082 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:38.670187 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:38.670409 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:38.670410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:38.670410 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:38.670671 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:38.670860 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:38.670932 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 03:58:38.671114 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":72,"threads_started":1}
I20260927 03:58:38.679157 20238 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:58:38.679762 20238 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:58:38.695609 20238 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d19ec021c9694877a39c8d0d8e5124fd: Generated new TSK 0
I20260927 03:58:38.695876 20238 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.185.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab.
Entry for principal kudu/127.15.185.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab.
I20260927 03:58:38.723855 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.65:0
--local_ip_for_outbound_sockets=127.15.185.65
--webserver_interface=127.15.185.65
--webserver_port=0
--tserver_master_addrs=127.15.185.126:45225
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab
--principal=kudu/127.15.185.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:40343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:58:38.807670 20252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:38.807883 20252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:38.807958 20252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:38.809121 20252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:38.809165 20252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:38.809191 20252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.65
I20260927 03:58:38.810770 20252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:40343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab
--principal=kudu/127.15.185.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.185.65
--webserver_port=0
--tserver_master_addrs=127.15.185.126:45225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.65
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:38.810976 20252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:38.811223 20252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:38.816105 20260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:38.816067 20257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:38.816076 20258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:38.816339 20252 server_base.cc:1034] running on GCE node
I20260927 03:58:38.816548 20252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:38.816790 20252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:38.817934 20252 hybrid_clock.cc:648] HybridClock initialized: now 1790481518817919 us; error 29 us; skew 500 ppm
Sep 27 03:58:38 dist-test-slave-glgz krb5kdc[20146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481518, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:38.820605 20252 init.cc:377] Logged in from keytab as kudu/127.15.185.65@KRBTEST.COM (short username kudu)
I20260927 03:58:38.820976 20252 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:38.821026 20252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:38.821118 20252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:38.822109 20252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/ts-0/data/instance:
uuid: "bb39f59f2de8418f9b9001e331f7d6dc"
format_stamp: "Formatted at 2026-09-27 03:58:38 on dist-test-slave-glgz"
server_key: "f653b108dff5276f8d359a9ea85f08e7"
server_key_iv: "014e89992025e3812f038aed7472d13d"
server_key_version: "encryptionkey@0"
I20260927 03:58:38.822417 20252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/ts-0/wal/instance:
uuid: "bb39f59f2de8418f9b9001e331f7d6dc"
format_stamp: "Formatted at 2026-09-27 03:58:38 on dist-test-slave-glgz"
server_key: "f653b108dff5276f8d359a9ea85f08e7"
server_key_iv: "014e89992025e3812f038aed7472d13d"
server_key_version: "encryptionkey@0"
I20260927 03:58:38.823721 20252 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:58:38.824455 20266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:38.824633 20252 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:58:38.824738 20252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/ts-0/wal
uuid: "bb39f59f2de8418f9b9001e331f7d6dc"
format_stamp: "Formatted at 2026-09-27 03:58:38 on dist-test-slave-glgz"
server_key: "f653b108dff5276f8d359a9ea85f08e7"
server_key_iv: "014e89992025e3812f038aed7472d13d"
server_key_version: "encryptionkey@0"
I20260927 03:58:38.824813 20252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:38.842123 20252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:38.844645 20252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:38.844815 20252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:38.845063 20252 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:58:38.845371 20252 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:58:38.845424 20252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:38.845469 20252 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:58:38.845495 20252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:38.851332 20252 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.65:46301
I20260927 03:58:38.851364 20379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.65:46301 every 8 connection(s)
I20260927 03:58:38.851639 20252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
Sep 27 03:58:38 dist-test-slave-glgz krb5kdc[20146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481518, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.65@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 03:58:38.858728 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 20252
I20260927 03:58:38.858872 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/ts-0/wal/instance
I20260927 03:58:38.859097 16101 external_mini_cluster.cc:1428] Setting key dc799b22f5df0d45a71fb0b4827522cd
I20260927 03:58:38.861120 20248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:38.852341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:36899 (local address 127.15.185.126:45225)
0927 03:58:38.852410 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:38.852411 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:38.852662 (+ 251us) server_negotiation.cc:407] Connection header received
0927 03:58:38.853117 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:38.853119 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:38.853132 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:38.853195 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:38.854185 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:38.854710 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:38.855776 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:38.856025 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:38.856086 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:38.858114 (+ 2028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:38.858115 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:38.858115 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:38.858124 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:38.859725 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:38.860141 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:38.860142 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:38.860142 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:38.860201 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:38.860478 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:38.860479 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:38.860479 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:38.860597 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:38.860668 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:38.860941 (+ 273us) server_negotiation.cc:299] Negotiation successful
0927 03:58:38.861032 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 03:58:38.861305 20382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:38.852312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45225 (local address 127.15.185.65:36899)
0927 03:58:38.852505 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:38.852548 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:38.853027 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:38.853268 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:38.853272 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:38.853529 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:38.854087 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:38.854092 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:38.854788 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:38.854789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:38.855695 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:38.855698 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:38.856105 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:38.856106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:38.856204 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:38.856655 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:38.856664 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:38.858026 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:38.859788 (+ 1762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:38.859794 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:38.859796 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:38.860022 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:38.860302 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:38.860302 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:38.860303 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:38.860407 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:38.860669 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:38.860676 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:38.860811 (+ 135us) client_negotiation.cc:769] Sending connection context
0927 03:58:38.860921 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 03:58:38.861075 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":118,"threads_started":1}
I20260927 03:58:38.861794 20380 heartbeater.cc:347] Connected to a master server at 127.15.185.126:45225
I20260927 03:58:38.861876 20380 heartbeater.cc:464] Registering TS with master...
I20260927 03:58:38.862044 20380 heartbeater.cc:511] Master 127.15.185.126:45225 requested a full tablet report, sending...
I20260927 03:58:38.862622 20190 ts_manager.cc:194] Registered new tserver with Master: bb39f59f2de8418f9b9001e331f7d6dc (127.15.185.65:46301)
I20260927 03:58:38.863590 20190 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.185.65@KRBTEST.COM'} at 127.15.185.65:36899
WARNING: no policy specified for kudu/127.15.185.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab.
Entry for principal kudu/127.15.185.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab.
I20260927 03:58:38.885325 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.66:0
--local_ip_for_outbound_sockets=127.15.185.66
--webserver_interface=127.15.185.66
--webserver_port=0
--tserver_master_addrs=127.15.185.126:45225
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab
--principal=kudu/127.15.185.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:40343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:58:38.966743 20385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:38.966965 20385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:38.966989 20385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:38.968204 20385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:38.968248 20385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:38.968274 20385 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.66
I20260927 03:58:38.969810 20385 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:40343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab
--principal=kudu/127.15.185.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.185.66
--webserver_port=0
--tserver_master_addrs=127.15.185.126:45225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20385
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.66
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:38.970070 20385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:38.970312 20385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:38.975340 20390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:38.975453 20391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:38.975340 20393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:38.975613 20385 server_base.cc:1034] running on GCE node
I20260927 03:58:38.975835 20385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:38.976090 20385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:38.977252 20385 hybrid_clock.cc:648] HybridClock initialized: now 1790481518977229 us; error 39 us; skew 500 ppm
Sep 27 03:58:38 dist-test-slave-glgz krb5kdc[20146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481518, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:38.980581 20385 init.cc:377] Logged in from keytab as kudu/127.15.185.66@KRBTEST.COM (short username kudu)
I20260927 03:58:38.981070 20385 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:38.981158 20385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:38.981298 20385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:38.982792 20385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/ts-1/data/instance:
uuid: "b9cf59d3c0d040879ffc7b9f392d82dc"
format_stamp: "Formatted at 2026-09-27 03:58:38 on dist-test-slave-glgz"
server_key: "0fc43b2f37ca9adab0e64b1a97505e91"
server_key_iv: "d0c36c87483e9c877b1435411b4869da"
server_key_version: "encryptionkey@0"
I20260927 03:58:38.983275 20385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/ts-1/wal/instance:
uuid: "b9cf59d3c0d040879ffc7b9f392d82dc"
format_stamp: "Formatted at 2026-09-27 03:58:38 on dist-test-slave-glgz"
server_key: "0fc43b2f37ca9adab0e64b1a97505e91"
server_key_iv: "d0c36c87483e9c877b1435411b4869da"
server_key_version: "encryptionkey@0"
I20260927 03:58:38.985204 20385 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260927 03:58:38.986313 20399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:38.986526 20385 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:58:38.986614 20385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/ts-1/wal
uuid: "b9cf59d3c0d040879ffc7b9f392d82dc"
format_stamp: "Formatted at 2026-09-27 03:58:38 on dist-test-slave-glgz"
server_key: "0fc43b2f37ca9adab0e64b1a97505e91"
server_key_iv: "d0c36c87483e9c877b1435411b4869da"
server_key_version: "encryptionkey@0"
I20260927 03:58:38.986680 20385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:39.007911 20385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:39.010509 20385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:39.010658 20385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:39.010914 20385 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:58:39.011340 20385 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:58:39.011418 20385 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:39.011462 20385 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:58:39.011494 20385 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:39.019238 20385 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.66:46447
I20260927 03:58:39.019274 20512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.66:46447 every 8 connection(s)
I20260927 03:58:39.019578 20385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
I20260927 03:58:39.020293 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 20385
I20260927 03:58:39.020418 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/ts-1/wal/instance
I20260927 03:58:39.020634 16101 external_mini_cluster.cc:1428] Setting key 25ee11051de0b0f09acc6130bd7a74bb
Sep 27 03:58:39 dist-test-slave-glgz krb5kdc[20146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481518, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.66@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.185.67@KRBTEST.COM; defaulting to no policy
I20260927 03:58:39.031513 20248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:39.020945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:54851 (local address 127.15.185.126:45225)
0927 03:58:39.021022 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:39.021024 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:39.021033 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:58:39.021689 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:39.021690 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:39.021701 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:39.021765 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:39.023230 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:39.024079 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:39.025220 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:39.025489 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:39.025551 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:39.028201 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:39.028202 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:39.028203 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:39.028212 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:39.030057 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:39.030489 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:39.030490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:39.030491 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:39.030550 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:39.030801 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:39.030802 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:39.030802 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:39.030931 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:39.031009 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:39.031320 (+ 311us) server_negotiation.cc:299] Negotiation successful
0927 03:58:39.031414 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 03:58:39.031720 20515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:39.020529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45225 (local address 127.15.185.66:54851)
0927 03:58:39.020771 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:39.020825 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:39.021566 (+ 741us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:39.021851 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:39.021857 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:39.022296 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:39.023101 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:39.023110 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:39.024220 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:39.024222 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:39.025127 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:39.025130 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:39.025590 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:39.025591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:39.025767 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:39.026324 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:39.026336 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:39.028098 (+ 1762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:39.030146 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:39.030153 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:39.030155 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:39.030379 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:39.030618 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:39.030619 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:39.030620 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:39.030732 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:39.031000 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:39.031006 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:39.031185 (+ 179us) client_negotiation.cc:769] Sending connection context
0927 03:58:39.031300 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 03:58:39.031478 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":85,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.67@KRBTEST.COM" created.
I20260927 03:58:39.032271 20513 heartbeater.cc:347] Connected to a master server at 127.15.185.126:45225
I20260927 03:58:39.032400 20513 heartbeater.cc:464] Registering TS with master...
I20260927 03:58:39.032557 20513 heartbeater.cc:511] Master 127.15.185.126:45225 requested a full tablet report, sending...
I20260927 03:58:39.032955 20190 ts_manager.cc:194] Registered new tserver with Master: b9cf59d3c0d040879ffc7b9f392d82dc (127.15.185.66:46447)
I20260927 03:58:39.033345 20190 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.185.66@KRBTEST.COM'} at 127.15.185.66:54851
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab.
Entry for principal kudu/127.15.185.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab.
I20260927 03:58:39.049590 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.67:0
--local_ip_for_outbound_sockets=127.15.185.67
--webserver_interface=127.15.185.67
--webserver_port=0
--tserver_master_addrs=127.15.185.126:45225
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab
--principal=kudu/127.15.185.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:40343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:58:39.131867 20518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:39.132081 20518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:39.132105 20518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:39.133296 20518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:39.133337 20518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:39.133389 20518 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.67
I20260927 03:58:39.134970 20518 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:40343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab
--principal=kudu/127.15.185.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.185.67
--webserver_port=0
--tserver_master_addrs=127.15.185.126:45225
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.67
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:39.135245 20518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:39.135497 20518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:39.140372 20524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:39.140379 20526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:39.140607 20518 server_base.cc:1034] running on GCE node
W20260927 03:58:39.140379 20523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:39.140879 20518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:39.141114 20518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:39.142270 20518 hybrid_clock.cc:648] HybridClock initialized: now 1790481519142254 us; error 26 us; skew 500 ppm
Sep 27 03:58:39 dist-test-slave-glgz krb5kdc[20146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481519, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:39.144986 20518 init.cc:377] Logged in from keytab as kudu/127.15.185.67@KRBTEST.COM (short username kudu)
I20260927 03:58:39.145408 20518 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:39.145494 20518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:39.145659 20518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:39.146936 20518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/ts-2/data/instance:
uuid: "f0c62b7fdc7f4a3bb6000f5fe5c89dfa"
format_stamp: "Formatted at 2026-09-27 03:58:39 on dist-test-slave-glgz"
server_key: "ad7b6b9c771dada76d89eb4a665c0a73"
server_key_iv: "2825ee7f17f03bcfb42c122b3e481456"
server_key_version: "encryptionkey@0"
I20260927 03:58:39.147574 20518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/ts-2/wal/instance:
uuid: "f0c62b7fdc7f4a3bb6000f5fe5c89dfa"
format_stamp: "Formatted at 2026-09-27 03:58:39 on dist-test-slave-glgz"
server_key: "ad7b6b9c771dada76d89eb4a665c0a73"
server_key_iv: "2825ee7f17f03bcfb42c122b3e481456"
server_key_version: "encryptionkey@0"
I20260927 03:58:39.149020 20518 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:58:39.149924 20532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:39.150146 20518 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:58:39.150250 20518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/ts-2/wal
uuid: "f0c62b7fdc7f4a3bb6000f5fe5c89dfa"
format_stamp: "Formatted at 2026-09-27 03:58:39 on dist-test-slave-glgz"
server_key: "ad7b6b9c771dada76d89eb4a665c0a73"
server_key_iv: "2825ee7f17f03bcfb42c122b3e481456"
server_key_version: "encryptionkey@0"
I20260927 03:58:39.150336 20518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:39.166080 20518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:39.168519 20518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:39.168699 20518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:39.169009 20518 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:58:39.169359 20518 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:58:39.169423 20518 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:39.169472 20518 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:58:39.169497 20518 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:39.175391 20518 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.67:33895
I20260927 03:58:39.175462 20645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.67:33895 every 8 connection(s)
I20260927 03:58:39.175745 20518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
Sep 27 03:58:39 dist-test-slave-glgz krb5kdc[20146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481519, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.67@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 03:58:39.184631 20248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:39.176535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:52069 (local address 127.15.185.126:45225)
0927 03:58:39.176625 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:39.176627 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:39.176769 (+ 142us) server_negotiation.cc:407] Connection header received
0927 03:58:39.177241 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:39.177242 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:39.177252 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:39.177321 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:39.178256 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:39.178756 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:39.179899 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:39.180089 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:39.180156 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:39.182267 (+ 2111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:39.182269 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:39.182269 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:39.182276 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:39.183532 (+ 1256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:39.183879 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:39.183880 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:39.183880 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:39.183933 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:39.184139 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:39.184140 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:39.184140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:39.184234 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:39.184282 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:39.184509 (+ 227us) server_negotiation.cc:299] Negotiation successful
0927 03:58:39.184567 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 03:58:39.184844 20648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:39.176495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45225 (local address 127.15.185.67:52069)
0927 03:58:39.176642 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:39.176684 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:39.177155 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:39.177384 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:39.177387 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:39.177651 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:39.178153 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:39.178158 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:39.178849 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:39.178850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:39.179813 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:39.179816 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:39.180196 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:39.180197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:39.180308 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:39.180789 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:39.180797 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:39.182171 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:39.183603 (+ 1432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:39.183608 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:39.183609 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:39.183815 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:39.183989 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:39.183990 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:39.183990 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:39.184083 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:39.184286 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:39.184290 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:39.184396 (+ 106us) client_negotiation.cc:769] Sending connection context
0927 03:58:39.184483 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 03:58:39.184596 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":77,"threads_started":1}
I20260927 03:58:39.184849 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 20518
I20260927 03:58:39.184932 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/ts-2/wal/instance
I20260927 03:58:39.185132 16101 external_mini_cluster.cc:1428] Setting key 875141b65d37878d47a3c1604c762059
I20260927 03:58:39.185316 20646 heartbeater.cc:347] Connected to a master server at 127.15.185.126:45225
I20260927 03:58:39.185422 20646 heartbeater.cc:464] Registering TS with master...
I20260927 03:58:39.185581 20646 heartbeater.cc:511] Master 127.15.185.126:45225 requested a full tablet report, sending...
I20260927 03:58:39.186019 20189 ts_manager.cc:194] Registered new tserver with Master: f0c62b7fdc7f4a3bb6000f5fe5c89dfa (127.15.185.67:33895)
I20260927 03:58:39.186424 20189 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.185.67@KRBTEST.COM'} at 127.15.185.67:52069
I20260927 03:58:39.196906 16101 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:58:39.204048 20248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:39.198171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40230 (local address 127.15.185.126:45225)
0927 03:58:39.198238 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:39.198240 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:39.198365 (+ 125us) server_negotiation.cc:407] Connection header received
0927 03:58:39.198419 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:39.198420 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:39.198429 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:39.198474 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:39.198994 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:39.199532 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:39.200230 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:39.200415 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:39.200466 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:39.201635 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:39.201636 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:39.201636 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:39.201642 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:39.203026 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:39.203377 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:39.203378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:39.203378 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:39.203441 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:39.203605 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:39.203606 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:39.203606 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:39.203737 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:39.203808 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:39.203899 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 03:58:39.203958 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 03:58:39.205816 20189 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40230:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:58:39.207389 20189 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:58:39.213853 20661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:39.209662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.66:46447 (local address 127.0.0.1:58154)
0927 03:58:39.210015 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:39.210025 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:39.210097 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:39.210383 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:39.210386 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:39.210388 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:58:39.210495 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:39.210497 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:39.211519 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:39.211520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:39.212715 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:39.212718 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:39.213525 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:39.213526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:39.213610 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:39.213624 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 03:58:39.213693 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 03:58:39.213744 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":84,"threads_started":1}
I20260927 03:58:39.213959 20657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:39.209624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.67:33895 (local address 127.0.0.1:44028)
0927 03:58:39.209892 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:39.210010 (+ 118us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:39.210096 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:39.210486 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:39.210487 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:39.210489 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:58:39.210559 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:39.210561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:39.211923 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:39.211924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:39.212864 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:39.212866 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:39.213702 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:39.213703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:39.213754 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:39.213756 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 03:58:39.213814 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 03:58:39.213856 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":64,"threads_started":1}
I20260927 03:58:39.213977 20659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:39.209726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44028 (local address 127.15.185.67:33895)
0927 03:58:39.209866 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:39.209872 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:39.210105 (+ 233us) server_negotiation.cc:407] Connection header received
0927 03:58:39.210191 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:39.210195 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:39.210213 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:39.210292 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:58:39.210650 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:39.211796 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:39.212932 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:39.213615 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:39.213691 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:39.213744 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:39.213844 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 03:58:39.213892 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":78,"threads_started":1}
I20260927 03:58:39.214201 20660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:39.209781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58154 (local address 127.15.185.66:46447)
0927 03:58:39.210242 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:39.210248 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:39.210256 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:58:39.210292 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:39.210294 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:39.210309 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:39.210381 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:58:39.210582 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:39.211416 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:39.212813 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:39.213440 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:39.213527 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:39.213587 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:39.214070 (+ 483us) server_negotiation.cc:299] Negotiation successful
0927 03:58:39.214110 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":85,"threads_started":1}
I20260927 03:58:39.214269 20658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:39.209827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.65:46301 (local address 127.0.0.1:43066)
0927 03:58:39.210204 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:39.210231 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:39.210269 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:39.210656 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:39.210657 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:39.210659 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:58:39.210690 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:39.210691 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:39.211970 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:39.211971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:39.213071 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:39.213073 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:39.214048 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:39.214050 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:39.214105 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:39.214108 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:58:39.214159 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 03:58:39.214192 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"mutex_wait_us":1,"thread_start_us":56,"threads_started":1}
I20260927 03:58:39.214481 20662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:39.210039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43066 (local address 127.15.185.65:46301)
0927 03:58:39.210423 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:39.210427 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:39.210435 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:58:39.210471 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:39.210484 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:39.210498 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:39.210745 (+ 247us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:58:39.210821 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:39.211867 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:39.213151 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:39.213855 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:39.213920 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:39.213967 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:39.214352 (+ 385us) server_negotiation.cc:299] Negotiation successful
0927 03:58:39.214390 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":75,"threads_started":1}
I20260927 03:58:39.214847 20447 tablet_service.cc:1467] Processing CreateTablet for tablet 9fdb9fc470794cc4a8e78d55348dd4e8 (DEFAULT_TABLE table=test-table [id=03eebd93bb5248b0a954aa8695ecfa1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:58:39.214967 20580 tablet_service.cc:1467] Processing CreateTablet for tablet 9fdb9fc470794cc4a8e78d55348dd4e8 (DEFAULT_TABLE table=test-table [id=03eebd93bb5248b0a954aa8695ecfa1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:58:39.215200 20447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fdb9fc470794cc4a8e78d55348dd4e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:39.215031 20314 tablet_service.cc:1467] Processing CreateTablet for tablet 9fdb9fc470794cc4a8e78d55348dd4e8 (DEFAULT_TABLE table=test-table [id=03eebd93bb5248b0a954aa8695ecfa1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:58:39.215402 20314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fdb9fc470794cc4a8e78d55348dd4e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:39.215404 20580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fdb9fc470794cc4a8e78d55348dd4e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:39.217605 20663 tablet_bootstrap.cc:492] T 9fdb9fc470794cc4a8e78d55348dd4e8 P b9cf59d3c0d040879ffc7b9f392d82dc: Bootstrap starting.
I20260927 03:58:39.218173 20663 tablet_bootstrap.cc:654] T 9fdb9fc470794cc4a8e78d55348dd4e8 P b9cf59d3c0d040879ffc7b9f392d82dc: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:39.218605 20665 tablet_bootstrap.cc:492] T 9fdb9fc470794cc4a8e78d55348dd4e8 P bb39f59f2de8418f9b9001e331f7d6dc: Bootstrap starting.
I20260927 03:58:39.218714 20663 log.cc:824] T 9fdb9fc470794cc4a8e78d55348dd4e8 P b9cf59d3c0d040879ffc7b9f392d82dc: Log is configured to *not* fsync() on all Append() calls
I20260927 03:58:39.219208 20664 tablet_bootstrap.cc:492] T 9fdb9fc470794cc4a8e78d55348dd4e8 P f0c62b7fdc7f4a3bb6000f5fe5c89dfa: Bootstrap starting.
I20260927 03:58:39.219292 20665 tablet_bootstrap.cc:654] T 9fdb9fc470794cc4a8e78d55348dd4e8 P bb39f59f2de8418f9b9001e331f7d6dc: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:39.219527 20663 tablet_bootstrap.cc:492] T 9fdb9fc470794cc4a8e78d55348dd4e8 P b9cf59d3c0d040879ffc7b9f392d82dc: No bootstrap required, opened a new log
I20260927 03:58:39.219615 20663 ts_tablet_manager.cc:1397] T 9fdb9fc470794cc4a8e78d55348dd4e8 P b9cf59d3c0d040879ffc7b9f392d82dc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:58:39.219879 20664 tablet_bootstrap.cc:654] T 9fdb9fc470794cc4a8e78d55348dd4e8 P f0c62b7fdc7f4a3bb6000f5fe5c89dfa: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:39.219877 20665 log.cc:824] T 9fdb9fc470794cc4a8e78d55348dd4e8 P bb39f59f2de8418f9b9001e331f7d6dc: Log is configured to *not* fsync() on all Append() calls
I20260927 03:58:39.220439 20664 log.cc:824] T 9fdb9fc470794cc4a8e78d55348dd4e8 P f0c62b7fdc7f4a3bb6000f5fe5c89dfa: Log is configured to *not* fsync() on all Append() calls
I20260927 03:58:39.220652 20665 tablet_bootstrap.cc:492] T 9fdb9fc470794cc4a8e78d55348dd4e8 P bb39f59f2de8418f9b9001e331f7d6dc: No bootstrap required, opened a new log
I20260927 03:58:39.220724 20665 ts_tablet_manager.cc:1397] T 9fdb9fc470794cc4a8e78d55348dd4e8 P bb39f59f2de8418f9b9001e331f7d6dc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:58:39.221272 20664 tablet_bootstrap.cc:492] T 9fdb9fc470794cc4a8e78d55348dd4e8 P f0c62b7fdc7f4a3bb6000f5fe5c89dfa: No bootstrap required, opened a new log
I20260927 03:58:39.221339 20664 ts_tablet_manager.cc:1397] T 9fdb9fc470794cc4a8e78d55348dd4e8 P f0c62b7fdc7f4a3bb6000f5fe5c89dfa: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:58:39.221624 20663 raft_consensus.cc:348] T 9fdb9fc470794cc4a8e78d55348dd4e8 P b9cf59d3c0d040879ffc7b9f392d82dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0c62b7fdc7f4a3bb6000f5fe5c89dfa" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 33895 } } peers { permanent_uuid: "bb39f59f2de8418f9b9001e331f7d6dc" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46301 } } peers { permanent_uuid: "b9cf59d3c0d040879ffc7b9f392d82dc" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 46447 } }
I20260927 03:58:39.221798 20663 raft_consensus.cc:374] T 9fdb9fc470794cc4a8e78d55348dd4e8 P b9cf59d3c0d040879ffc7b9f392d82dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:39.221843 20663 raft_consensus.cc:729] T 9fdb9fc470794cc4a8e78d55348dd4e8 P b9cf59d3c0d040879ffc7b9f392d82dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9cf59d3c0d040879ffc7b9f392d82dc, State: Initialized, Role: FOLLOWER
I20260927 03:58:39.221954 20663 consensus_queue.cc:260] T 9fdb9fc470794cc4a8e78d55348dd4e8 P b9cf59d3c0d040879ffc7b9f392d82dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0c62b7fdc7f4a3bb6000f5fe5c89dfa" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 33895 } } peers { permanent_uuid: "bb39f59f2de8418f9b9001e331f7d6dc" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46301 } } peers { permanent_uuid: "b9cf59d3c0d040879ffc7b9f392d82dc" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 46447 } }
I20260927 03:58:39.222252 20663 ts_tablet_manager.cc:1428] T 9fdb9fc470794cc4a8e78d55348dd4e8 P b9cf59d3c0d040879ffc7b9f392d82dc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:58:39.222380 20513 heartbeater.cc:503] Master 127.15.185.126:45225 was elected leader, sending a full tablet report...
I20260927 03:58:39.222657 20665 raft_consensus.cc:348] T 9fdb9fc470794cc4a8e78d55348dd4e8 P bb39f59f2de8418f9b9001e331f7d6dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0c62b7fdc7f4a3bb6000f5fe5c89dfa" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 33895 } } peers { permanent_uuid: "bb39f59f2de8418f9b9001e331f7d6dc" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46301 } } peers { permanent_uuid: "b9cf59d3c0d040879ffc7b9f392d82dc" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 46447 } }
I20260927 03:58:39.222822 20665 raft_consensus.cc:374] T 9fdb9fc470794cc4a8e78d55348dd4e8 P bb39f59f2de8418f9b9001e331f7d6dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:39.222855 20665 raft_consensus.cc:729] T 9fdb9fc470794cc4a8e78d55348dd4e8 P bb39f59f2de8418f9b9001e331f7d6dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb39f59f2de8418f9b9001e331f7d6dc, State: Initialized, Role: FOLLOWER
I20260927 03:58:39.223026 20665 consensus_queue.cc:260] T 9fdb9fc470794cc4a8e78d55348dd4e8 P bb39f59f2de8418f9b9001e331f7d6dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0c62b7fdc7f4a3bb6000f5fe5c89dfa" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 33895 } } peers { permanent_uuid: "bb39f59f2de8418f9b9001e331f7d6dc" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46301 } } peers { permanent_uuid: "b9cf59d3c0d040879ffc7b9f392d82dc" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 46447 } }
I20260927 03:58:39.223318 20380 heartbeater.cc:503] Master 127.15.185.126:45225 was elected leader, sending a full tablet report...
I20260927 03:58:39.223323 20665 ts_tablet_manager.cc:1428] T 9fdb9fc470794cc4a8e78d55348dd4e8 P bb39f59f2de8418f9b9001e331f7d6dc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:58:39.223311 20664 raft_consensus.cc:348] T 9fdb9fc470794cc4a8e78d55348dd4e8 P f0c62b7fdc7f4a3bb6000f5fe5c89dfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0c62b7fdc7f4a3bb6000f5fe5c89dfa" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 33895 } } peers { permanent_uuid: "bb39f59f2de8418f9b9001e331f7d6dc" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46301 } } peers { permanent_uuid: "b9cf59d3c0d040879ffc7b9f392d82dc" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 46447 } }
I20260927 03:58:39.223474 20664 raft_consensus.cc:374] T 9fdb9fc470794cc4a8e78d55348dd4e8 P f0c62b7fdc7f4a3bb6000f5fe5c89dfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:39.223533 20664 raft_consensus.cc:729] T 9fdb9fc470794cc4a8e78d55348dd4e8 P f0c62b7fdc7f4a3bb6000f5fe5c89dfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0c62b7fdc7f4a3bb6000f5fe5c89dfa, State: Initialized, Role: FOLLOWER
I20260927 03:58:39.223668 20664 consensus_queue.cc:260] T 9fdb9fc470794cc4a8e78d55348dd4e8 P f0c62b7fdc7f4a3bb6000f5fe5c89dfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0c62b7fdc7f4a3bb6000f5fe5c89dfa" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 33895 } } peers { permanent_uuid: "bb39f59f2de8418f9b9001e331f7d6dc" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46301 } } peers { permanent_uuid: "b9cf59d3c0d040879ffc7b9f392d82dc" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 46447 } }
I20260927 03:58:39.223953 20664 ts_tablet_manager.cc:1428] T 9fdb9fc470794cc4a8e78d55348dd4e8 P f0c62b7fdc7f4a3bb6000f5fe5c89dfa: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:58:39.223961 20646 heartbeater.cc:503] Master 127.15.185.126:45225 was elected leader, sending a full tablet report...
W20260927 03:58:39.270143 20514 tablet.cc:2369] T 9fdb9fc470794cc4a8e78d55348dd4e8 P b9cf59d3c0d040879ffc7b9f392d82dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:58:39.326594 20669 raft_consensus.cc:482] T 9fdb9fc470794cc4a8e78d55348dd4e8 P b9cf59d3c0d040879ffc7b9f392d82dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:58:39.326727 20669 raft_consensus.cc:504] T 9fdb9fc470794cc4a8e78d55348dd4e8 P b9cf59d3c0d040879ffc7b9f392d82dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0c62b7fdc7f4a3bb6000f5fe5c89dfa" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 33895 } } peers { permanent_uuid: "bb39f59f2de8418f9b9001e331f7d6dc" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46301 } } peers { permanent_uuid: "b9cf59d3c0d040879ffc7b9f392d82dc" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 46447 } }
I20260927 03:58:39.327190 20669 leader_election.cc:288] T 9fdb9fc470794cc4a8e78d55348dd4e8 P b9cf59d3c0d040879ffc7b9f392d82dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0c62b7fdc7f4a3bb6000f5fe5c89dfa (127.15.185.67:33895), bb39f59f2de8418f9b9001e331f7d6dc (127.15.185.65:46301)
I20260927 03:58:39.330730 20662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:39.327606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:44493 (local address 127.15.185.65:46301)
0927 03:58:39.327651 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:39.327653 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:39.327749 (+ 96us) server_negotiation.cc:407] Connection header received
0927 03:58:39.327794 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:39.327795 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:39.327804 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:39.327873 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:58:39.328046 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:39.328825 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:39.329956 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:39.330363 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:39.330455 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:39.330464 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:39.330617 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 03:58:39.330656 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 03:58:39.330749 20659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:39.327690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:59069 (local address 127.15.185.67:33895)
0927 03:58:39.327771 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:39.327774 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:39.327782 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:58:39.327805 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:39.327806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:39.327818 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:39.327874 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:58:39.328046 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:39.328837 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:39.329934 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:39.330398 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:39.330485 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:39.330494 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:39.330646 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 03:58:39.330681 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 03:58:39.330777 20515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:39.327404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.67:33895 (local address 127.15.185.66:59069)
0927 03:58:39.327489 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:39.327502 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:39.327601 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:39.327911 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:39.327911 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:39.327912 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:58:39.327968 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:39.327970 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:39.328924 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:39.328924 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:39.329844 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:39.329847 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:39.330478 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:39.330479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:39.330561 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:39.330570 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 03:58:39.330640 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 03:58:39.330675 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 03:58:39.330977 20672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:39.327514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.65:46301 (local address 127.15.185.66:44493)
0927 03:58:39.327666 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:39.327677 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:39.327741 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:39.327882 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:39.327883 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:39.327885 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:58:39.327969 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:39.327970 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:39.328973 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:39.328974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:39.329850 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:39.329852 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:39.330454 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:39.330455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:39.330531 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:39.330534 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:58:39.330608 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 03:58:39.330647 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":1,"thread_start_us":102,"threads_started":1}
I20260927 03:58:39.331156 20600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fdb9fc470794cc4a8e78d55348dd4e8" candidate_uuid: "b9cf59d3c0d040879ffc7b9f392d82dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0c62b7fdc7f4a3bb6000f5fe5c89dfa" is_pre_election: true
I20260927 03:58:39.331332 20600 raft_consensus.cc:2454] T 9fdb9fc470794cc4a8e78d55348dd4e8 P f0c62b7fdc7f4a3bb6000f5fe5c89dfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9cf59d3c0d040879ffc7b9f392d82dc in term 0.
I20260927 03:58:39.331568 20401 leader_election.cc:302] T 9fdb9fc470794cc4a8e78d55348dd4e8 P b9cf59d3c0d040879ffc7b9f392d82dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9cf59d3c0d040879ffc7b9f392d82dc, f0c62b7fdc7f4a3bb6000f5fe5c89dfa; no voters:
I20260927 03:58:39.331694 20669 raft_consensus.cc:2781] T 9fdb9fc470794cc4a8e78d55348dd4e8 P b9cf59d3c0d040879ffc7b9f392d82dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:58:39.331674 20334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fdb9fc470794cc4a8e78d55348dd4e8" candidate_uuid: "b9cf59d3c0d040879ffc7b9f392d82dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb39f59f2de8418f9b9001e331f7d6dc" is_pre_election: true
I20260927 03:58:39.331759 20669 raft_consensus.cc:482] T 9fdb9fc470794cc4a8e78d55348dd4e8 P b9cf59d3c0d040879ffc7b9f392d82dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:58:39.331784 20669 raft_consensus.cc:3037] T 9fdb9fc470794cc4a8e78d55348dd4e8 P b9cf59d3c0d040879ffc7b9f392d82dc [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:39.331821 20334 raft_consensus.cc:2454] T 9fdb9fc470794cc4a8e78d55348dd4e8 P bb39f59f2de8418f9b9001e331f7d6dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9cf59d3c0d040879ffc7b9f392d82dc in term 0.
I20260927 03:58:39.332554 20669 raft_consensus.cc:504] T 9fdb9fc470794cc4a8e78d55348dd4e8 P b9cf59d3c0d040879ffc7b9f392d82dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0c62b7fdc7f4a3bb6000f5fe5c89dfa" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 33895 } } peers { permanent_uuid: "bb39f59f2de8418f9b9001e331f7d6dc" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46301 } } peers { permanent_uuid: "b9cf59d3c0d040879ffc7b9f392d82dc" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 46447 } }
I20260927 03:58:39.332685 20669 leader_election.cc:288] T 9fdb9fc470794cc4a8e78d55348dd4e8 P b9cf59d3c0d040879ffc7b9f392d82dc [CANDIDATE]: Term 1 election: Requested vote from peers f0c62b7fdc7f4a3bb6000f5fe5c89dfa (127.15.185.67:33895), bb39f59f2de8418f9b9001e331f7d6dc (127.15.185.65:46301)
I20260927 03:58:39.332898 20600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fdb9fc470794cc4a8e78d55348dd4e8" candidate_uuid: "b9cf59d3c0d040879ffc7b9f392d82dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0c62b7fdc7f4a3bb6000f5fe5c89dfa"
I20260927 03:58:39.332870 20334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fdb9fc470794cc4a8e78d55348dd4e8" candidate_uuid: "b9cf59d3c0d040879ffc7b9f392d82dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb39f59f2de8418f9b9001e331f7d6dc"
I20260927 03:58:39.332964 20600 raft_consensus.cc:3037] T 9fdb9fc470794cc4a8e78d55348dd4e8 P f0c62b7fdc7f4a3bb6000f5fe5c89dfa [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:39.332988 20334 raft_consensus.cc:3037] T 9fdb9fc470794cc4a8e78d55348dd4e8 P bb39f59f2de8418f9b9001e331f7d6dc [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:39.333634 20600 raft_consensus.cc:2454] T 9fdb9fc470794cc4a8e78d55348dd4e8 P f0c62b7fdc7f4a3bb6000f5fe5c89dfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9cf59d3c0d040879ffc7b9f392d82dc in term 1.
I20260927 03:58:39.333783 20401 leader_election.cc:302] T 9fdb9fc470794cc4a8e78d55348dd4e8 P b9cf59d3c0d040879ffc7b9f392d82dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9cf59d3c0d040879ffc7b9f392d82dc, f0c62b7fdc7f4a3bb6000f5fe5c89dfa; no voters:
I20260927 03:58:39.333798 20334 raft_consensus.cc:2454] T 9fdb9fc470794cc4a8e78d55348dd4e8 P bb39f59f2de8418f9b9001e331f7d6dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9cf59d3c0d040879ffc7b9f392d82dc in term 1.
I20260927 03:58:39.333892 20669 raft_consensus.cc:2781] T 9fdb9fc470794cc4a8e78d55348dd4e8 P b9cf59d3c0d040879ffc7b9f392d82dc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:58:39.334048 20669 raft_consensus.cc:686] T 9fdb9fc470794cc4a8e78d55348dd4e8 P b9cf59d3c0d040879ffc7b9f392d82dc [term 1 LEADER]: Becoming Leader. State: Replica: b9cf59d3c0d040879ffc7b9f392d82dc, State: Running, Role: LEADER
I20260927 03:58:39.334151 20669 consensus_queue.cc:237] T 9fdb9fc470794cc4a8e78d55348dd4e8 P b9cf59d3c0d040879ffc7b9f392d82dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0c62b7fdc7f4a3bb6000f5fe5c89dfa" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 33895 } } peers { permanent_uuid: "bb39f59f2de8418f9b9001e331f7d6dc" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46301 } } peers { permanent_uuid: "b9cf59d3c0d040879ffc7b9f392d82dc" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 46447 } }
I20260927 03:58:39.334924 20190 catalog_manager.cc:5495] T 9fdb9fc470794cc4a8e78d55348dd4e8 P b9cf59d3c0d040879ffc7b9f392d82dc reported cstate change: term changed from 0 to 1, leader changed from <none> to b9cf59d3c0d040879ffc7b9f392d82dc (127.15.185.66). New cstate: current_term: 1 leader_uuid: "b9cf59d3c0d040879ffc7b9f392d82dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0c62b7fdc7f4a3bb6000f5fe5c89dfa" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 33895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb39f59f2de8418f9b9001e331f7d6dc" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9cf59d3c0d040879ffc7b9f392d82dc" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 46447 } health_report { overall_health: HEALTHY } } }
W20260927 03:58:39.352308 20381 tablet.cc:2369] T 9fdb9fc470794cc4a8e78d55348dd4e8 P bb39f59f2de8418f9b9001e331f7d6dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:58:39.362670 20660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:39.359973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40958 (local address 127.15.185.66:46447)
0927 03:58:39.360052 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:39.360054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:39.360062 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:58:39.360120 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:39.360121 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:39.360137 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:39.360197 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:58:39.360397 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:39.360877 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:39.361763 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:39.361987 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:39.362072 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:39.362278 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:58:39.362396 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:58:39.362514 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:39.362533 (+ 19us) server_negotiation.cc:299] Negotiation successful
0927 03:58:39.362554 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 03:58:39.364616 20600 raft_consensus.cc:1261] T 9fdb9fc470794cc4a8e78d55348dd4e8 P f0c62b7fdc7f4a3bb6000f5fe5c89dfa [term 1 FOLLOWER]: Refusing update from remote peer b9cf59d3c0d040879ffc7b9f392d82dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:58:39.365022 20334 raft_consensus.cc:1261] T 9fdb9fc470794cc4a8e78d55348dd4e8 P bb39f59f2de8418f9b9001e331f7d6dc [term 1 FOLLOWER]: Refusing update from remote peer b9cf59d3c0d040879ffc7b9f392d82dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:58:39.365028 20673 consensus_queue.cc:1035] T 9fdb9fc470794cc4a8e78d55348dd4e8 P b9cf59d3c0d040879ffc7b9f392d82dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0c62b7fdc7f4a3bb6000f5fe5c89dfa" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 33895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:58:39.365352 20673 consensus_queue.cc:1035] T 9fdb9fc470794cc4a8e78d55348dd4e8 P b9cf59d3c0d040879ffc7b9f392d82dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb39f59f2de8418f9b9001e331f7d6dc" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:58:39.368007 20678 mvcc.cc:205] Tried to move back new op lower bound from 7333812303314620416 to 7333812303193149440. Current Snapshot: MvccSnapshot[applied={T|T < 7333812303314620416}]
I20260927 03:58:39.368439 20677 mvcc.cc:205] Tried to move back new op lower bound from 7333812303314620416 to 7333812303193149440. Current Snapshot: MvccSnapshot[applied={T|T < 7333812303314620416 or (T in {7333812303314620416})}]
I20260927 03:58:39.369666 20190 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40230:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:58:39.369777 20190 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40230:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:58:39.370528 20190 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d19ec021c9694877a39c8d0d8e5124fd: Sending DeleteTablet for 3 replicas of tablet 9fdb9fc470794cc4a8e78d55348dd4e8
I20260927 03:58:39.370854 20447 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fdb9fc470794cc4a8e78d55348dd4e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:58:39 UTC) from {username='kudu', principal='kudu/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:58154
I20260927 03:58:39.370878 20314 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fdb9fc470794cc4a8e78d55348dd4e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:58:39 UTC) from {username='kudu', principal='kudu/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:43066
I20260927 03:58:39.371088 20580 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fdb9fc470794cc4a8e78d55348dd4e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:58:39 UTC) from {username='kudu', principal='kudu/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:44028
I20260927 03:58:39.371150 20686 tablet_replica.cc:330] T 9fdb9fc470794cc4a8e78d55348dd4e8 P b9cf59d3c0d040879ffc7b9f392d82dc: stopping tablet replica
I20260927 03:58:39.371248 20686 raft_consensus.cc:2229] T 9fdb9fc470794cc4a8e78d55348dd4e8 P b9cf59d3c0d040879ffc7b9f392d82dc [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:58:39.371330 20688 tablet_replica.cc:330] T 9fdb9fc470794cc4a8e78d55348dd4e8 P f0c62b7fdc7f4a3bb6000f5fe5c89dfa: stopping tablet replica
I20260927 03:58:39.371351 20686 raft_consensus.cc:2258] T 9fdb9fc470794cc4a8e78d55348dd4e8 P b9cf59d3c0d040879ffc7b9f392d82dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:58:39.371511 20687 tablet_replica.cc:330] T 9fdb9fc470794cc4a8e78d55348dd4e8 P bb39f59f2de8418f9b9001e331f7d6dc: stopping tablet replica
I20260927 03:58:39.371603 20687 raft_consensus.cc:2229] T 9fdb9fc470794cc4a8e78d55348dd4e8 P bb39f59f2de8418f9b9001e331f7d6dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:58:39.371699 20687 raft_consensus.cc:2258] T 9fdb9fc470794cc4a8e78d55348dd4e8 P bb39f59f2de8418f9b9001e331f7d6dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:58:39.371752 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 20252
I20260927 03:58:39.372622 20686 ts_tablet_manager.cc:1905] T 9fdb9fc470794cc4a8e78d55348dd4e8 P b9cf59d3c0d040879ffc7b9f392d82dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:58:39.374259 20686 ts_tablet_manager.cc:1918] T 9fdb9fc470794cc4a8e78d55348dd4e8 P b9cf59d3c0d040879ffc7b9f392d82dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:58:39.374334 20686 log.cc:1196] T 9fdb9fc470794cc4a8e78d55348dd4e8 P b9cf59d3c0d040879ffc7b9f392d82dc: Deleting WAL directory at /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/ts-1/wal/wals/9fdb9fc470794cc4a8e78d55348dd4e8
I20260927 03:58:39.374604 20686 ts_tablet_manager.cc:1939] T 9fdb9fc470794cc4a8e78d55348dd4e8 P b9cf59d3c0d040879ffc7b9f392d82dc: Deleting consensus metadata
I20260927 03:58:39.374727 20688 raft_consensus.cc:2229] T 9fdb9fc470794cc4a8e78d55348dd4e8 P f0c62b7fdc7f4a3bb6000f5fe5c89dfa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:58:39.374847 20688 raft_consensus.cc:2258] T 9fdb9fc470794cc4a8e78d55348dd4e8 P f0c62b7fdc7f4a3bb6000f5fe5c89dfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:58:39.375370 20176 catalog_manager.cc:4841] TS b9cf59d3c0d040879ffc7b9f392d82dc (127.15.185.66:46447): tablet 9fdb9fc470794cc4a8e78d55348dd4e8 (table test-table [id=03eebd93bb5248b0a954aa8695ecfa1e]) successfully deleted
I20260927 03:58:39.375505 20688 ts_tablet_manager.cc:1905] T 9fdb9fc470794cc4a8e78d55348dd4e8 P f0c62b7fdc7f4a3bb6000f5fe5c89dfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:58:39.377113 20688 ts_tablet_manager.cc:1918] T 9fdb9fc470794cc4a8e78d55348dd4e8 P f0c62b7fdc7f4a3bb6000f5fe5c89dfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:58:39.377188 20688 log.cc:1196] T 9fdb9fc470794cc4a8e78d55348dd4e8 P f0c62b7fdc7f4a3bb6000f5fe5c89dfa: Deleting WAL directory at /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481498018407-16101-0/minicluster-data/ts-2/wal/wals/9fdb9fc470794cc4a8e78d55348dd4e8
I20260927 03:58:39.377481 20688 ts_tablet_manager.cc:1939] T 9fdb9fc470794cc4a8e78d55348dd4e8 P f0c62b7fdc7f4a3bb6000f5fe5c89dfa: Deleting consensus metadata
W20260927 03:58:39.377579 20175 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.185.65:46301 (error 108)
I20260927 03:58:39.377944 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 20385
I20260927 03:58:39.378013 20175 catalog_manager.cc:4841] TS f0c62b7fdc7f4a3bb6000f5fe5c89dfa (127.15.185.67:33895): tablet 9fdb9fc470794cc4a8e78d55348dd4e8 (table test-table [id=03eebd93bb5248b0a954aa8695ecfa1e]) successfully deleted
I20260927 03:58:39.378113 20658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:39.377965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.65:46301 (local address 127.0.0.1:50492)
0927 03:58:39.378026 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:39.378073 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.65:46301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 03:58:39.378626 20175 catalog_manager.cc:4584] TS bb39f59f2de8418f9b9001e331f7d6dc (127.15.185.65:46301): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9fdb9fc470794cc4a8e78d55348dd4e8: Network error: Client connection negotiation failed: client connection to 127.15.185.65:46301: connect: Connection refused (error 111)
I20260927 03:58:39.382349 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 20518
I20260927 03:58:39.387017 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 20159
2026-09-27T03:58:39Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2011 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-27T03:58:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:58:39Z Disabled control of system clock
I20260927 03:58:39.414005 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:37255
--webserver_interface=127.15.185.126
--webserver_port=0
--builtin_ntp_servers=127.15.185.84:34623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.185.126:37255
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 03:58:39.494212 20693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:39.494416 20693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:39.494436 20693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:39.495610 20693 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:58:39.495651 20693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:39.495662 20693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:39.495675 20693 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:58:39.495687 20693 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:58:39.497016 20693 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:34623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.185.126:37255
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:37255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:39.497269 20693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:39.497623 20693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:39.502413 20698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:39.502429 20699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:39.502545 20693 server_base.cc:1034] running on GCE node
W20260927 03:58:39.502424 20701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:39.502827 20693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:39.503175 20693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:39.504360 20693 hybrid_clock.cc:648] HybridClock initialized: now 1790481519504343 us; error 37 us; skew 500 ppm
I20260927 03:58:39.505549 20693 webserver.cc:460] Webserver started at http://127.15.185.126:45875/ using document root <none> and password file <none>
I20260927 03:58:39.505779 20693 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:39.505844 20693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:39.505980 20693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:39.507028 20693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/master-0/data/instance:
uuid: "8b94ab60b68f4abc9969b97e03a912c3"
format_stamp: "Formatted at 2026-09-27 03:58:39 on dist-test-slave-glgz"
server_key: "c6ca906c4f0b984c81f6f1a597805483"
server_key_iv: "cb0c938dcc0e23272b8fbb365fa04155"
server_key_version: "encryptionkey@0"
I20260927 03:58:39.507478 20693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/master-0/wal/instance:
uuid: "8b94ab60b68f4abc9969b97e03a912c3"
format_stamp: "Formatted at 2026-09-27 03:58:39 on dist-test-slave-glgz"
server_key: "c6ca906c4f0b984c81f6f1a597805483"
server_key_iv: "cb0c938dcc0e23272b8fbb365fa04155"
server_key_version: "encryptionkey@0"
I20260927 03:58:39.508747 20693 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:58:39.509534 20707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:39.509714 20693 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:58:39.509809 20693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/master-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/master-0/wal
uuid: "8b94ab60b68f4abc9969b97e03a912c3"
format_stamp: "Formatted at 2026-09-27 03:58:39 on dist-test-slave-glgz"
server_key: "c6ca906c4f0b984c81f6f1a597805483"
server_key_iv: "cb0c938dcc0e23272b8fbb365fa04155"
server_key_version: "encryptionkey@0"
I20260927 03:58:39.509884 20693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:39.513195 20693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:39.513458 20693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:39.513594 20693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:39.517251 20693 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.126:37255
I20260927 03:58:39.517298 20759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.126:37255 every 8 connection(s)
I20260927 03:58:39.517661 20693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
I20260927 03:58:39.517938 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 20693
I20260927 03:58:39.518040 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/master-0/wal/instance
I20260927 03:58:39.518194 16101 external_mini_cluster.cc:1428] Setting key ece0ba466521b266abdcdb8fbdaa7ea9
I20260927 03:58:39.518447 20760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:39.521131 20763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:39.518666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44044 (local address 127.15.185.126:37255)
0927 03:58:39.519014 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:39.519025 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:39.519098 (+ 73us) server_negotiation.cc:407] Connection header received
0927 03:58:39.519662 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:39.519672 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:39.519735 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:39.519895 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:39.520657 (+ 762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:39.520661 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:39.520661 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:58:39.520665 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:39.520713 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:58:39.520727 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:39.520808 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:39.520861 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 03:58:39.520926 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":110,"threads_started":1}
I20260927 03:58:39.522092 20760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b94ab60b68f4abc9969b97e03a912c3: Bootstrap starting.
I20260927 03:58:39.522621 20760 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b94ab60b68f4abc9969b97e03a912c3: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:39.522837 20760 log.cc:824] T 00000000000000000000000000000000 P 8b94ab60b68f4abc9969b97e03a912c3: Log is configured to *not* fsync() on all Append() calls
I20260927 03:58:39.523516 20760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b94ab60b68f4abc9969b97e03a912c3: No bootstrap required, opened a new log
I20260927 03:58:39.524741 20760 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b94ab60b68f4abc9969b97e03a912c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b94ab60b68f4abc9969b97e03a912c3" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 37255 } }
I20260927 03:58:39.524866 20760 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b94ab60b68f4abc9969b97e03a912c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:39.524892 20760 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b94ab60b68f4abc9969b97e03a912c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b94ab60b68f4abc9969b97e03a912c3, State: Initialized, Role: FOLLOWER
I20260927 03:58:39.525013 20760 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b94ab60b68f4abc9969b97e03a912c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b94ab60b68f4abc9969b97e03a912c3" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 37255 } }
I20260927 03:58:39.525094 20760 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b94ab60b68f4abc9969b97e03a912c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:58:39.525115 20760 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b94ab60b68f4abc9969b97e03a912c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:58:39.525141 20760 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b94ab60b68f4abc9969b97e03a912c3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:39.525591 20760 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b94ab60b68f4abc9969b97e03a912c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b94ab60b68f4abc9969b97e03a912c3" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 37255 } }
I20260927 03:58:39.525668 20760 leader_election.cc:302] T 00000000000000000000000000000000 P 8b94ab60b68f4abc9969b97e03a912c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b94ab60b68f4abc9969b97e03a912c3; no voters:
I20260927 03:58:39.525787 20760 leader_election.cc:288] T 00000000000000000000000000000000 P 8b94ab60b68f4abc9969b97e03a912c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:58:39.526081 20765 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b94ab60b68f4abc9969b97e03a912c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:58:39.526085 20760 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b94ab60b68f4abc9969b97e03a912c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:58:39.526226 20765 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b94ab60b68f4abc9969b97e03a912c3 [term 1 LEADER]: Becoming Leader. State: Replica: 8b94ab60b68f4abc9969b97e03a912c3, State: Running, Role: LEADER
I20260927 03:58:39.526342 20765 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b94ab60b68f4abc9969b97e03a912c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b94ab60b68f4abc9969b97e03a912c3" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 37255 } }
I20260927 03:58:39.526706 20767 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b94ab60b68f4abc9969b97e03a912c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b94ab60b68f4abc9969b97e03a912c3. Latest consensus state: current_term: 1 leader_uuid: "8b94ab60b68f4abc9969b97e03a912c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b94ab60b68f4abc9969b97e03a912c3" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 37255 } } }
I20260927 03:58:39.526726 20766 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b94ab60b68f4abc9969b97e03a912c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b94ab60b68f4abc9969b97e03a912c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b94ab60b68f4abc9969b97e03a912c3" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 37255 } } }
I20260927 03:58:39.526849 20767 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b94ab60b68f4abc9969b97e03a912c3 [sys.catalog]: This master's current role is: LEADER
I20260927 03:58:39.526858 20766 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b94ab60b68f4abc9969b97e03a912c3 [sys.catalog]: This master's current role is: LEADER
I20260927 03:58:39.527194 20774 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:58:39.527830 20774 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:58:39.529171 20774 catalog_manager.cc:1347] Generated new cluster ID: 89bac20029414bd19e58a0c8141ed3d9
I20260927 03:58:39.529227 20774 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:58:39.535488 20774 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:58:39.535641 20774 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:58:39.543218 20774 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b94ab60b68f4abc9969b97e03a912c3: Generated new TSK 0
I20260927 03:58:39.543392 20774 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:58:39.547228 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.65:0
--local_ip_for_outbound_sockets=127.15.185.65
--webserver_interface=127.15.185.65
--webserver_port=0
--tserver_master_addrs=127.15.185.126:37255
--builtin_ntp_servers=127.15.185.84:34623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 03:58:39.636263 20784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:39.636472 20784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:39.636495 20784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:39.637650 20784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:39.637687 20784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:39.637712 20784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.65
I20260927 03:58:39.639006 20784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:34623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.185.65
--webserver_port=0
--tserver_master_addrs=127.15.185.126:37255
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.65
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:39.639297 20784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:39.639604 20784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:39.645483 20790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:39.645483 20792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:39.645853 20784 server_base.cc:1034] running on GCE node
W20260927 03:58:39.645542 20789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:39.646126 20784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:39.646395 20784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:39.647569 20784 hybrid_clock.cc:648] HybridClock initialized: now 1790481519647550 us; error 32 us; skew 500 ppm
I20260927 03:58:39.649006 20784 webserver.cc:460] Webserver started at http://127.15.185.65:39227/ using document root <none> and password file <none>
I20260927 03:58:39.649261 20784 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:39.649325 20784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:39.649459 20784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:39.650864 20784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/ts-0/data/instance:
uuid: "7662bf33e835431398984502168ba1e4"
format_stamp: "Formatted at 2026-09-27 03:58:39 on dist-test-slave-glgz"
server_key: "ffb1e2151bb3e2f0fd3273d56d50533b"
server_key_iv: "605677174d4886892b16a4deba2f2ea7"
server_key_version: "encryptionkey@0"
I20260927 03:58:39.651273 20784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/ts-0/wal/instance:
uuid: "7662bf33e835431398984502168ba1e4"
format_stamp: "Formatted at 2026-09-27 03:58:39 on dist-test-slave-glgz"
server_key: "ffb1e2151bb3e2f0fd3273d56d50533b"
server_key_iv: "605677174d4886892b16a4deba2f2ea7"
server_key_version: "encryptionkey@0"
I20260927 03:58:39.652792 20784 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:58:39.653671 20798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:39.653884 20784 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:58:39.654001 20784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/ts-0/wal
uuid: "7662bf33e835431398984502168ba1e4"
format_stamp: "Formatted at 2026-09-27 03:58:39 on dist-test-slave-glgz"
server_key: "ffb1e2151bb3e2f0fd3273d56d50533b"
server_key_iv: "605677174d4886892b16a4deba2f2ea7"
server_key_version: "encryptionkey@0"
I20260927 03:58:39.654096 20784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:39.657269 20784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:39.657519 20784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:39.657655 20784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:39.657853 20784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:58:39.658174 20784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:58:39.658228 20784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:39.658267 20784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:58:39.658295 20784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:39.663748 20784 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.65:38547
I20260927 03:58:39.663787 20911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.65:38547 every 8 connection(s)
I20260927 03:58:39.664202 20784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
I20260927 03:58:39.667335 20763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:39.665016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:56367 (local address 127.15.185.126:37255)
0927 03:58:39.665067 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:39.665069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:39.665295 (+ 226us) server_negotiation.cc:407] Connection header received
0927 03:58:39.665725 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:39.665726 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:39.665732 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:39.665813 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:39.666974 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:39.666976 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:39.666976 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:58:39.666984 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:39.667004 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:58:39.667009 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:39.667121 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:39.667247 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 03:58:39.667263 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 03:58:39.667498 20914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:39.664995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:37255 (local address 127.15.185.65:56367)
0927 03:58:39.665169 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:39.665207 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:39.665645 (+ 438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:39.665863 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:39.665866 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:39.665874 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:39.666824 (+ 950us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:58:39.666824 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:39.666852 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:58:39.666881 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:39.667112 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:39.667120 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:39.667124 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:58:39.667213 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 03:58:39.667280 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":85,"threads_started":1}
I20260927 03:58:39.667981 20912 heartbeater.cc:347] Connected to a master server at 127.15.185.126:37255
I20260927 03:58:39.668087 20912 heartbeater.cc:464] Registering TS with master...
I20260927 03:58:39.668228 20912 heartbeater.cc:511] Master 127.15.185.126:37255 requested a full tablet report, sending...
I20260927 03:58:39.668731 20724 ts_manager.cc:194] Registered new tserver with Master: 7662bf33e835431398984502168ba1e4 (127.15.185.65:38547)
I20260927 03:58:39.672577 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 20784
I20260927 03:58:39.672708 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/ts-0/wal/instance
I20260927 03:58:39.672994 16101 external_mini_cluster.cc:1428] Setting key d59bc83f3199c8dad71859ff477a7911
I20260927 03:58:39.673907 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.66:0
--local_ip_for_outbound_sockets=127.15.185.66
--webserver_interface=127.15.185.66
--webserver_port=0
--tserver_master_addrs=127.15.185.126:37255
--builtin_ntp_servers=127.15.185.84:34623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 03:58:39.754045 20915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:39.754267 20915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:39.754289 20915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:39.755404 20915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:39.755438 20915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:39.755462 20915 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.66
I20260927 03:58:39.756886 20915 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:34623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.185.66
--webserver_port=0
--tserver_master_addrs=127.15.185.126:37255
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20915
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.66
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:39.757090 20915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:39.757336 20915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:39.762766 20923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:39.762765 20920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:39.762773 20921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:39.762825 20915 server_base.cc:1034] running on GCE node
I20260927 03:58:39.763186 20915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:39.763406 20915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:39.764567 20915 hybrid_clock.cc:648] HybridClock initialized: now 1790481519764555 us; error 27 us; skew 500 ppm
I20260927 03:58:39.766031 20915 webserver.cc:460] Webserver started at http://127.15.185.66:45063/ using document root <none> and password file <none>
I20260927 03:58:39.766319 20915 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:39.766372 20915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:39.766582 20915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:39.767843 20915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/ts-1/data/instance:
uuid: "322e31085562442ea93349fc84078086"
format_stamp: "Formatted at 2026-09-27 03:58:39 on dist-test-slave-glgz"
server_key: "1f4e3672515fc4599b15c6d819bbf3b2"
server_key_iv: "55773c0db9754081f7a83e44019960c4"
server_key_version: "encryptionkey@0"
I20260927 03:58:39.768270 20915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/ts-1/wal/instance:
uuid: "322e31085562442ea93349fc84078086"
format_stamp: "Formatted at 2026-09-27 03:58:39 on dist-test-slave-glgz"
server_key: "1f4e3672515fc4599b15c6d819bbf3b2"
server_key_iv: "55773c0db9754081f7a83e44019960c4"
server_key_version: "encryptionkey@0"
I20260927 03:58:39.769707 20915 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 03:58:39.770439 20929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:39.770613 20915 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:58:39.770717 20915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/ts-1/wal
uuid: "322e31085562442ea93349fc84078086"
format_stamp: "Formatted at 2026-09-27 03:58:39 on dist-test-slave-glgz"
server_key: "1f4e3672515fc4599b15c6d819bbf3b2"
server_key_iv: "55773c0db9754081f7a83e44019960c4"
server_key_version: "encryptionkey@0"
I20260927 03:58:39.770797 20915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:39.774089 20915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:39.774354 20915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:39.774498 20915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:39.774698 20915 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:58:39.775000 20915 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:58:39.775075 20915 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:39.775104 20915 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:58:39.775136 20915 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:39.781318 20915 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.66:37749
I20260927 03:58:39.781409 21042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.66:37749 every 8 connection(s)
I20260927 03:58:39.781736 20915 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
I20260927 03:58:39.785205 20763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:39.782687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:49153 (local address 127.15.185.126:37255)
0927 03:58:39.782778 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:39.782781 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:39.782982 (+ 201us) server_negotiation.cc:407] Connection header received
0927 03:58:39.783567 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:39.783569 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:39.783575 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:39.783627 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:39.784898 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:39.784907 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:39.784907 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:58:39.784912 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:39.784935 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:58:39.784939 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:39.785012 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:39.785115 (+ 103us) server_negotiation.cc:299] Negotiation successful
0927 03:58:39.785130 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 03:58:39.785439 21045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:39.782554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:37255 (local address 127.15.185.66:49153)
0927 03:58:39.782825 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:39.782876 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:39.783454 (+ 578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:39.783703 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:39.783707 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:39.783717 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:39.784730 (+ 1013us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:58:39.784730 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:39.784760 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:58:39.784791 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:39.785010 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:39.785017 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:39.785020 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:58:39.785127 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 03:58:39.785192 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":111,"threads_started":1}
I20260927 03:58:39.786108 21043 heartbeater.cc:347] Connected to a master server at 127.15.185.126:37255
I20260927 03:58:39.786208 21043 heartbeater.cc:464] Registering TS with master...
I20260927 03:58:39.786329 21043 heartbeater.cc:511] Master 127.15.185.126:37255 requested a full tablet report, sending...
I20260927 03:58:39.786705 20724 ts_manager.cc:194] Registered new tserver with Master: 322e31085562442ea93349fc84078086 (127.15.185.66:37749)
I20260927 03:58:39.788499 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 20915
I20260927 03:58:39.788585 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/ts-1/wal/instance
I20260927 03:58:39.788753 16101 external_mini_cluster.cc:1428] Setting key 35641c587b75ee73b13fecf23391d998
I20260927 03:58:39.789685 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.67:0
--local_ip_for_outbound_sockets=127.15.185.67
--webserver_interface=127.15.185.67
--webserver_port=0
--tserver_master_addrs=127.15.185.126:37255
--builtin_ntp_servers=127.15.185.84:34623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 03:58:39.874567 21046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:39.874784 21046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:39.874809 21046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:39.876071 21046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:39.876121 21046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:39.876148 21046 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.67
I20260927 03:58:39.877476 21046 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:34623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.185.67
--webserver_port=0
--tserver_master_addrs=127.15.185.126:37255
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.67
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:39.877681 21046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:39.877936 21046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:39.882941 21054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:39.882942 21051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:39.883124 21046 server_base.cc:1034] running on GCE node
W20260927 03:58:39.882975 21052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:39.883450 21046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:39.883700 21046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:39.884859 21046 hybrid_clock.cc:648] HybridClock initialized: now 1790481519884855 us; error 39 us; skew 500 ppm
I20260927 03:58:39.886215 21046 webserver.cc:460] Webserver started at http://127.15.185.67:41807/ using document root <none> and password file <none>
I20260927 03:58:39.886471 21046 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:39.886543 21046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:39.886682 21046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:39.887801 21046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/ts-2/data/instance:
uuid: "234b4e84873d4de3addba45bff5cb20d"
format_stamp: "Formatted at 2026-09-27 03:58:39 on dist-test-slave-glgz"
server_key: "10f26a4dbd61a4943933e8da789e3c50"
server_key_iv: "13d74defaa0b3d02bc80a6addc885805"
server_key_version: "encryptionkey@0"
I20260927 03:58:39.888157 21046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/ts-2/wal/instance:
uuid: "234b4e84873d4de3addba45bff5cb20d"
format_stamp: "Formatted at 2026-09-27 03:58:39 on dist-test-slave-glgz"
server_key: "10f26a4dbd61a4943933e8da789e3c50"
server_key_iv: "13d74defaa0b3d02bc80a6addc885805"
server_key_version: "encryptionkey@0"
I20260927 03:58:39.889483 21046 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:58:39.890275 21060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:39.890443 21046 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:58:39.890550 21046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/ts-2/wal
uuid: "234b4e84873d4de3addba45bff5cb20d"
format_stamp: "Formatted at 2026-09-27 03:58:39 on dist-test-slave-glgz"
server_key: "10f26a4dbd61a4943933e8da789e3c50"
server_key_iv: "13d74defaa0b3d02bc80a6addc885805"
server_key_version: "encryptionkey@0"
I20260927 03:58:39.890625 21046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:39.894114 21046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:39.894387 21046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:39.894522 21046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:39.894711 21046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:58:39.895015 21046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:58:39.895082 21046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:39.895123 21046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:58:39.895153 21046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:39.900938 21046 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.67:36897
I20260927 03:58:39.901014 21173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.67:36897 every 8 connection(s)
I20260927 03:58:39.901331 21046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
I20260927 03:58:39.904346 20763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:39.902087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:37003 (local address 127.15.185.126:37255)
0927 03:58:39.902136 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:39.902139 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:39.902338 (+ 199us) server_negotiation.cc:407] Connection header received
0927 03:58:39.902848 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:39.902849 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:39.902857 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:39.902923 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:39.904098 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:39.904101 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:39.904101 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:58:39.904106 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:39.904119 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:58:39.904123 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:39.904179 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:39.904260 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 03:58:39.904274 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 03:58:39.904547 21176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:39.902048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:37255 (local address 127.15.185.67:37003)
0927 03:58:39.902211 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:39.902251 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:39.902739 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:39.902992 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:39.902996 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:39.903006 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:39.903947 (+ 941us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:58:39.903948 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:39.903977 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:58:39.904004 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:39.904187 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:39.904192 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:39.904194 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 03:58:39.904261 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 03:58:39.904320 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":61,"threads_started":1}
I20260927 03:58:39.904604 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 21046
I20260927 03:58:39.904702 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/ts-2/wal/instance
I20260927 03:58:39.904902 16101 external_mini_cluster.cc:1428] Setting key 3ad84067974b8ebe1319c2f052b4167a
I20260927 03:58:39.905058 21174 heartbeater.cc:347] Connected to a master server at 127.15.185.126:37255
I20260927 03:58:39.905151 21174 heartbeater.cc:464] Registering TS with master...
I20260927 03:58:39.905296 21174 heartbeater.cc:511] Master 127.15.185.126:37255 requested a full tablet report, sending...
I20260927 03:58:39.905751 20724 ts_manager.cc:194] Registered new tserver with Master: 234b4e84873d4de3addba45bff5cb20d (127.15.185.67:36897)
I20260927 03:58:39.916625 16101 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:58:39.919188 20763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:39.918025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44050 (local address 127.15.185.126:37255)
0927 03:58:39.918109 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:39.918112 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:39.918217 (+ 105us) server_negotiation.cc:407] Connection header received
0927 03:58:39.918320 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:39.918321 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:39.918328 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:39.918380 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:39.918956 (+ 576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:39.918957 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:39.918958 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:58:39.918962 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:39.918983 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:58:39.918987 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:39.919035 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:39.919097 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 03:58:39.919112 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 03:58:39.920756 20724 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44050:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:58:39.922293 20724 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:58:39.925993 21183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:39.924353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.66:37749 (local address 127.0.0.1:44822)
0927 03:58:39.924646 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:39.924657 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:39.924708 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:39.924938 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:39.924944 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:39.924953 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:39.925413 (+ 460us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:58:39.925413 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:39.925430 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:58:39.925454 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:39.925828 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:39.925833 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:39.925837 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:58:39.925904 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 03:58:39.925918 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":2,"thread_start_us":74,"threads_started":1}
I20260927 03:58:39.926017 21185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:39.924455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44822 (local address 127.15.185.66:37749)
0927 03:58:39.924771 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:39.924776 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:39.924787 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:58:39.924815 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:39.924818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:39.924827 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:39.924906 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:39.925698 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:39.925703 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:39.925704 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:58:39.925708 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:39.925736 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:58:39.925751 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:39.925832 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:39.925917 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 03:58:39.925933 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":102,"threads_started":1}
I20260927 03:58:39.926265 21187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:39.924745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37928 (local address 127.15.185.67:36897)
0927 03:58:39.924931 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:39.924936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:39.924945 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:58:39.924978 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:39.924980 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:39.924989 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:39.925070 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:39.925711 (+ 641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:39.925716 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:39.925717 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:58:39.925720 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:39.925746 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:58:39.925763 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:39.925832 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:39.926134 (+ 302us) server_negotiation.cc:299] Negotiation successful
0927 03:58:39.926151 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":95,"threads_started":1}
I20260927 03:58:39.926595 21186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:39.924543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.67:36897 (local address 127.0.0.1:37928)
0927 03:58:39.924656 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:39.924665 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:39.924708 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:39.925207 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:39.925208 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:39.925211 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:39.925612 (+ 401us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:58:39.925613 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:39.925620 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:58:39.925634 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:39.925829 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:39.925835 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:39.925839 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:58:39.925909 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 03:58:39.925922 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":77,"threads_started":1}
I20260927 03:58:39.926695 20977 tablet_service.cc:1467] Processing CreateTablet for tablet f1b9c87258e74f388b620e5da0ff06a5 (DEFAULT_TABLE table=test-table [id=88f162ae699048ea8368f8319bcb2567]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:58:39.927055 20977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1b9c87258e74f388b620e5da0ff06a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:39.927103 21182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:39.924353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.65:38547 (local address 127.0.0.1:48056)
0927 03:58:39.924538 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:39.924579 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:39.924632 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:39.926092 (+ 1460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:39.926093 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:39.926096 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:39.926559 (+ 463us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:58:39.926559 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:39.926565 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:58:39.926575 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:39.926941 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:39.926942 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:39.926942 (+ 0us) client_negotiation.cc:769] Sending connection context
0927 03:58:39.926993 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 03:58:39.927015 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":54,"threads_started":1}
I20260927 03:58:39.927145 21184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:39.924455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48056 (local address 127.15.185.65:38547)
0927 03:58:39.925921 (+ 1466us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:39.925925 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:39.925934 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:58:39.925957 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:39.925960 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:39.925968 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:39.926388 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:39.926808 (+ 420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:39.926812 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:39.926812 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:58:39.926816 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:39.926846 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:58:39.926861 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:39.926942 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:39.927011 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 03:58:39.927026 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1425,"thread_start_us":101,"threads_started":1}
I20260927 03:58:39.927405 21108 tablet_service.cc:1467] Processing CreateTablet for tablet f1b9c87258e74f388b620e5da0ff06a5 (DEFAULT_TABLE table=test-table [id=88f162ae699048ea8368f8319bcb2567]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:58:39.927717 21108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1b9c87258e74f388b620e5da0ff06a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:39.927994 20846 tablet_service.cc:1467] Processing CreateTablet for tablet f1b9c87258e74f388b620e5da0ff06a5 (DEFAULT_TABLE table=test-table [id=88f162ae699048ea8368f8319bcb2567]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:58:39.928366 20846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1b9c87258e74f388b620e5da0ff06a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:39.930398 21188 tablet_bootstrap.cc:492] T f1b9c87258e74f388b620e5da0ff06a5 P 322e31085562442ea93349fc84078086: Bootstrap starting.
I20260927 03:58:39.930811 21189 tablet_bootstrap.cc:492] T f1b9c87258e74f388b620e5da0ff06a5 P 234b4e84873d4de3addba45bff5cb20d: Bootstrap starting.
I20260927 03:58:39.931177 21190 tablet_bootstrap.cc:492] T f1b9c87258e74f388b620e5da0ff06a5 P 7662bf33e835431398984502168ba1e4: Bootstrap starting.
I20260927 03:58:39.931205 21188 tablet_bootstrap.cc:654] T f1b9c87258e74f388b620e5da0ff06a5 P 322e31085562442ea93349fc84078086: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:39.931514 21189 tablet_bootstrap.cc:654] T f1b9c87258e74f388b620e5da0ff06a5 P 234b4e84873d4de3addba45bff5cb20d: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:39.931551 21188 log.cc:824] T f1b9c87258e74f388b620e5da0ff06a5 P 322e31085562442ea93349fc84078086: Log is configured to *not* fsync() on all Append() calls
I20260927 03:58:39.931807 21190 tablet_bootstrap.cc:654] T f1b9c87258e74f388b620e5da0ff06a5 P 7662bf33e835431398984502168ba1e4: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:39.931824 21189 log.cc:824] T f1b9c87258e74f388b620e5da0ff06a5 P 234b4e84873d4de3addba45bff5cb20d: Log is configured to *not* fsync() on all Append() calls
I20260927 03:58:39.932126 21190 log.cc:824] T f1b9c87258e74f388b620e5da0ff06a5 P 7662bf33e835431398984502168ba1e4: Log is configured to *not* fsync() on all Append() calls
I20260927 03:58:39.932368 21188 tablet_bootstrap.cc:492] T f1b9c87258e74f388b620e5da0ff06a5 P 322e31085562442ea93349fc84078086: No bootstrap required, opened a new log
I20260927 03:58:39.932464 21188 ts_tablet_manager.cc:1397] T f1b9c87258e74f388b620e5da0ff06a5 P 322e31085562442ea93349fc84078086: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:58:39.932829 21190 tablet_bootstrap.cc:492] T f1b9c87258e74f388b620e5da0ff06a5 P 7662bf33e835431398984502168ba1e4: No bootstrap required, opened a new log
I20260927 03:58:39.932901 21190 ts_tablet_manager.cc:1397] T f1b9c87258e74f388b620e5da0ff06a5 P 7662bf33e835431398984502168ba1e4: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 03:58:39.935185 21188 raft_consensus.cc:348] T f1b9c87258e74f388b620e5da0ff06a5 P 322e31085562442ea93349fc84078086 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7662bf33e835431398984502168ba1e4" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 38547 } } peers { permanent_uuid: "322e31085562442ea93349fc84078086" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 37749 } } peers { permanent_uuid: "234b4e84873d4de3addba45bff5cb20d" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36897 } }
I20260927 03:58:39.935364 21188 raft_consensus.cc:374] T f1b9c87258e74f388b620e5da0ff06a5 P 322e31085562442ea93349fc84078086 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:39.935398 21188 raft_consensus.cc:729] T f1b9c87258e74f388b620e5da0ff06a5 P 322e31085562442ea93349fc84078086 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 322e31085562442ea93349fc84078086, State: Initialized, Role: FOLLOWER
I20260927 03:58:39.935551 21188 consensus_queue.cc:260] T f1b9c87258e74f388b620e5da0ff06a5 P 322e31085562442ea93349fc84078086 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7662bf33e835431398984502168ba1e4" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 38547 } } peers { permanent_uuid: "322e31085562442ea93349fc84078086" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 37749 } } peers { permanent_uuid: "234b4e84873d4de3addba45bff5cb20d" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36897 } }
I20260927 03:58:39.935482 21190 raft_consensus.cc:348] T f1b9c87258e74f388b620e5da0ff06a5 P 7662bf33e835431398984502168ba1e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7662bf33e835431398984502168ba1e4" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 38547 } } peers { permanent_uuid: "322e31085562442ea93349fc84078086" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 37749 } } peers { permanent_uuid: "234b4e84873d4de3addba45bff5cb20d" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36897 } }
I20260927 03:58:39.935616 21189 tablet_bootstrap.cc:492] T f1b9c87258e74f388b620e5da0ff06a5 P 234b4e84873d4de3addba45bff5cb20d: No bootstrap required, opened a new log
I20260927 03:58:39.935631 21190 raft_consensus.cc:374] T f1b9c87258e74f388b620e5da0ff06a5 P 7662bf33e835431398984502168ba1e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:39.935670 21190 raft_consensus.cc:729] T f1b9c87258e74f388b620e5da0ff06a5 P 7662bf33e835431398984502168ba1e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7662bf33e835431398984502168ba1e4, State: Initialized, Role: FOLLOWER
I20260927 03:58:39.935695 21189 ts_tablet_manager.cc:1397] T f1b9c87258e74f388b620e5da0ff06a5 P 234b4e84873d4de3addba45bff5cb20d: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260927 03:58:39.935804 21188 ts_tablet_manager.cc:1428] T f1b9c87258e74f388b620e5da0ff06a5 P 322e31085562442ea93349fc84078086: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260927 03:58:39.935804 21190 consensus_queue.cc:260] T f1b9c87258e74f388b620e5da0ff06a5 P 7662bf33e835431398984502168ba1e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7662bf33e835431398984502168ba1e4" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 38547 } } peers { permanent_uuid: "322e31085562442ea93349fc84078086" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 37749 } } peers { permanent_uuid: "234b4e84873d4de3addba45bff5cb20d" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36897 } }
I20260927 03:58:39.935917 21043 heartbeater.cc:503] Master 127.15.185.126:37255 was elected leader, sending a full tablet report...
I20260927 03:58:39.936046 21190 ts_tablet_manager.cc:1428] T f1b9c87258e74f388b620e5da0ff06a5 P 7662bf33e835431398984502168ba1e4: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260927 03:58:39.936139 20912 heartbeater.cc:503] Master 127.15.185.126:37255 was elected leader, sending a full tablet report...
I20260927 03:58:39.937925 21189 raft_consensus.cc:348] T f1b9c87258e74f388b620e5da0ff06a5 P 234b4e84873d4de3addba45bff5cb20d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7662bf33e835431398984502168ba1e4" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 38547 } } peers { permanent_uuid: "322e31085562442ea93349fc84078086" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 37749 } } peers { permanent_uuid: "234b4e84873d4de3addba45bff5cb20d" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36897 } }
I20260927 03:58:39.938068 21189 raft_consensus.cc:374] T f1b9c87258e74f388b620e5da0ff06a5 P 234b4e84873d4de3addba45bff5cb20d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:39.938112 21189 raft_consensus.cc:729] T f1b9c87258e74f388b620e5da0ff06a5 P 234b4e84873d4de3addba45bff5cb20d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 234b4e84873d4de3addba45bff5cb20d, State: Initialized, Role: FOLLOWER
I20260927 03:58:39.938226 21189 consensus_queue.cc:260] T f1b9c87258e74f388b620e5da0ff06a5 P 234b4e84873d4de3addba45bff5cb20d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7662bf33e835431398984502168ba1e4" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 38547 } } peers { permanent_uuid: "322e31085562442ea93349fc84078086" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 37749 } } peers { permanent_uuid: "234b4e84873d4de3addba45bff5cb20d" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36897 } }
I20260927 03:58:39.938457 21189 ts_tablet_manager.cc:1428] T f1b9c87258e74f388b620e5da0ff06a5 P 234b4e84873d4de3addba45bff5cb20d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:58:39.938525 21174 heartbeater.cc:503] Master 127.15.185.126:37255 was elected leader, sending a full tablet report...
W20260927 03:58:40.032368 21044 tablet.cc:2369] T f1b9c87258e74f388b620e5da0ff06a5 P 322e31085562442ea93349fc84078086: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:58:40.081858 21196 raft_consensus.cc:482] T f1b9c87258e74f388b620e5da0ff06a5 P 234b4e84873d4de3addba45bff5cb20d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:58:40.081984 21196 raft_consensus.cc:504] T f1b9c87258e74f388b620e5da0ff06a5 P 234b4e84873d4de3addba45bff5cb20d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7662bf33e835431398984502168ba1e4" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 38547 } } peers { permanent_uuid: "322e31085562442ea93349fc84078086" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 37749 } } peers { permanent_uuid: "234b4e84873d4de3addba45bff5cb20d" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36897 } }
I20260927 03:58:40.082434 21196 leader_election.cc:288] T f1b9c87258e74f388b620e5da0ff06a5 P 234b4e84873d4de3addba45bff5cb20d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7662bf33e835431398984502168ba1e4 (127.15.185.65:38547), 322e31085562442ea93349fc84078086 (127.15.185.66:37749)
I20260927 03:58:40.083738 21176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:40.082655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.65:38547 (local address 127.15.185.67:45101)
0927 03:58:40.082695 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:40.082708 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:40.082748 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:40.082915 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:40.082917 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:40.082919 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:40.083412 (+ 493us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:58:40.083413 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:40.083419 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:58:40.083430 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:40.083579 (+ 149us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:40.083580 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:40.083580 (+ 0us) client_negotiation.cc:769] Sending connection context
0927 03:58:40.083633 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 03:58:40.083661 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260927 03:58:40.083730 21184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:40.082719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:45101 (local address 127.15.185.65:38547)
0927 03:58:40.082774 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:40.082776 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:40.082791 (+ 15us) server_negotiation.cc:407] Connection header received
0927 03:58:40.082821 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:40.082823 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:40.082828 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:40.082887 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:40.083488 (+ 601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:40.083489 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:40.083490 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:58:40.083494 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:40.083513 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:58:40.083518 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:40.083576 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:40.083640 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 03:58:40.083655 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 03:58:40.084017 21197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:40.082660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.66:37749 (local address 127.15.185.67:56475)
0927 03:58:40.083027 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:40.083036 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:40.083151 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:40.083281 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:40.083281 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:40.083284 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:40.083690 (+ 406us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:58:40.083691 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:40.083696 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:58:40.083705 (+ 9us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:40.083890 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:40.083891 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:40.083892 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 03:58:40.083939 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 03:58:40.083950 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":143,"thread_start_us":109,"threads_started":1}
I20260927 03:58:40.084039 21185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:40.082739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:56475 (local address 127.15.185.66:37749)
0927 03:58:40.082782 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:40.082784 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:40.083130 (+ 346us) server_negotiation.cc:407] Connection header received
0927 03:58:40.083220 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:40.083221 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:40.083226 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:40.083290 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:40.083791 (+ 501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:40.083792 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:40.083793 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:58:40.083796 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:40.083813 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:58:40.083817 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:40.083874 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:40.083960 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 03:58:40.083973 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 03:58:40.084134 20866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1b9c87258e74f388b620e5da0ff06a5" candidate_uuid: "234b4e84873d4de3addba45bff5cb20d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7662bf33e835431398984502168ba1e4" is_pre_election: true
I20260927 03:58:40.084261 20866 raft_consensus.cc:2454] T f1b9c87258e74f388b620e5da0ff06a5 P 7662bf33e835431398984502168ba1e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 234b4e84873d4de3addba45bff5cb20d in term 0.
I20260927 03:58:40.084482 21061 leader_election.cc:302] T f1b9c87258e74f388b620e5da0ff06a5 P 234b4e84873d4de3addba45bff5cb20d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 234b4e84873d4de3addba45bff5cb20d, 7662bf33e835431398984502168ba1e4; no voters:
I20260927 03:58:40.084460 20997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1b9c87258e74f388b620e5da0ff06a5" candidate_uuid: "234b4e84873d4de3addba45bff5cb20d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "322e31085562442ea93349fc84078086" is_pre_election: true
I20260927 03:58:40.084595 20997 raft_consensus.cc:2454] T f1b9c87258e74f388b620e5da0ff06a5 P 322e31085562442ea93349fc84078086 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 234b4e84873d4de3addba45bff5cb20d in term 0.
I20260927 03:58:40.084601 21196 raft_consensus.cc:2781] T f1b9c87258e74f388b620e5da0ff06a5 P 234b4e84873d4de3addba45bff5cb20d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:58:40.084669 21196 raft_consensus.cc:482] T f1b9c87258e74f388b620e5da0ff06a5 P 234b4e84873d4de3addba45bff5cb20d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:58:40.084703 21196 raft_consensus.cc:3037] T f1b9c87258e74f388b620e5da0ff06a5 P 234b4e84873d4de3addba45bff5cb20d [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:40.085418 21196 raft_consensus.cc:504] T f1b9c87258e74f388b620e5da0ff06a5 P 234b4e84873d4de3addba45bff5cb20d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7662bf33e835431398984502168ba1e4" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 38547 } } peers { permanent_uuid: "322e31085562442ea93349fc84078086" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 37749 } } peers { permanent_uuid: "234b4e84873d4de3addba45bff5cb20d" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36897 } }
I20260927 03:58:40.085572 21196 leader_election.cc:288] T f1b9c87258e74f388b620e5da0ff06a5 P 234b4e84873d4de3addba45bff5cb20d [CANDIDATE]: Term 1 election: Requested vote from peers 7662bf33e835431398984502168ba1e4 (127.15.185.65:38547), 322e31085562442ea93349fc84078086 (127.15.185.66:37749)
I20260927 03:58:40.085742 20866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1b9c87258e74f388b620e5da0ff06a5" candidate_uuid: "234b4e84873d4de3addba45bff5cb20d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7662bf33e835431398984502168ba1e4"
I20260927 03:58:40.085748 20997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1b9c87258e74f388b620e5da0ff06a5" candidate_uuid: "234b4e84873d4de3addba45bff5cb20d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "322e31085562442ea93349fc84078086"
I20260927 03:58:40.085830 20866 raft_consensus.cc:3037] T f1b9c87258e74f388b620e5da0ff06a5 P 7662bf33e835431398984502168ba1e4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:40.085846 20997 raft_consensus.cc:3037] T f1b9c87258e74f388b620e5da0ff06a5 P 322e31085562442ea93349fc84078086 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:40.086623 20997 raft_consensus.cc:2454] T f1b9c87258e74f388b620e5da0ff06a5 P 322e31085562442ea93349fc84078086 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 234b4e84873d4de3addba45bff5cb20d in term 1.
I20260927 03:58:40.086745 20866 raft_consensus.cc:2454] T f1b9c87258e74f388b620e5da0ff06a5 P 7662bf33e835431398984502168ba1e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 234b4e84873d4de3addba45bff5cb20d in term 1.
I20260927 03:58:40.086846 21062 leader_election.cc:302] T f1b9c87258e74f388b620e5da0ff06a5 P 234b4e84873d4de3addba45bff5cb20d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 234b4e84873d4de3addba45bff5cb20d, 322e31085562442ea93349fc84078086; no voters:
I20260927 03:58:40.086961 21196 raft_consensus.cc:2781] T f1b9c87258e74f388b620e5da0ff06a5 P 234b4e84873d4de3addba45bff5cb20d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:58:40.087180 21196 raft_consensus.cc:686] T f1b9c87258e74f388b620e5da0ff06a5 P 234b4e84873d4de3addba45bff5cb20d [term 1 LEADER]: Becoming Leader. State: Replica: 234b4e84873d4de3addba45bff5cb20d, State: Running, Role: LEADER
I20260927 03:58:40.087283 21196 consensus_queue.cc:237] T f1b9c87258e74f388b620e5da0ff06a5 P 234b4e84873d4de3addba45bff5cb20d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7662bf33e835431398984502168ba1e4" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 38547 } } peers { permanent_uuid: "322e31085562442ea93349fc84078086" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 37749 } } peers { permanent_uuid: "234b4e84873d4de3addba45bff5cb20d" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36897 } }
I20260927 03:58:40.088179 20723 catalog_manager.cc:5495] T f1b9c87258e74f388b620e5da0ff06a5 P 234b4e84873d4de3addba45bff5cb20d reported cstate change: term changed from 0 to 1, leader changed from <none> to 234b4e84873d4de3addba45bff5cb20d (127.15.185.67). New cstate: current_term: 1 leader_uuid: "234b4e84873d4de3addba45bff5cb20d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7662bf33e835431398984502168ba1e4" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 38547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "322e31085562442ea93349fc84078086" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 37749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "234b4e84873d4de3addba45bff5cb20d" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36897 } health_report { overall_health: HEALTHY } } }
I20260927 03:58:40.109344 21187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:40.108248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37940 (local address 127.15.185.67:36897)
0927 03:58:40.108324 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:40.108326 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:40.108340 (+ 14us) server_negotiation.cc:407] Connection header received
0927 03:58:40.108374 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:40.108375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:40.108388 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:40.108449 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:40.109091 (+ 642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:40.109093 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:40.109093 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:58:40.109098 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:40.109114 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:58:40.109118 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:40.109193 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:40.109249 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 03:58:40.109264 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 03:58:40.111322 20997 raft_consensus.cc:1261] T f1b9c87258e74f388b620e5da0ff06a5 P 322e31085562442ea93349fc84078086 [term 1 FOLLOWER]: Refusing update from remote peer 234b4e84873d4de3addba45bff5cb20d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:58:40.111322 20866 raft_consensus.cc:1261] T f1b9c87258e74f388b620e5da0ff06a5 P 7662bf33e835431398984502168ba1e4 [term 1 FOLLOWER]: Refusing update from remote peer 234b4e84873d4de3addba45bff5cb20d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:58:40.111690 21198 consensus_queue.cc:1035] T f1b9c87258e74f388b620e5da0ff06a5 P 234b4e84873d4de3addba45bff5cb20d [LEADER]: Connected to new peer: Peer: permanent_uuid: "322e31085562442ea93349fc84078086" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 37749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:58:40.111853 21196 consensus_queue.cc:1035] T f1b9c87258e74f388b620e5da0ff06a5 P 234b4e84873d4de3addba45bff5cb20d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7662bf33e835431398984502168ba1e4" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 38547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:58:40.115648 21203 mvcc.cc:205] Tried to move back new op lower bound from 7333812306372841472 to 7333812306278178816. Current Snapshot: MvccSnapshot[applied={T|T < 7333812306372841472 or (T in {7333812306372841472})}]
I20260927 03:58:40.117079 20724 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:58:40.117175 20724 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:58:40.117902 20724 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8b94ab60b68f4abc9969b97e03a912c3: Sending DeleteTablet for 3 replicas of tablet f1b9c87258e74f388b620e5da0ff06a5
I20260927 03:58:40.118232 20846 tablet_service.cc:1514] Processing DeleteTablet for tablet f1b9c87258e74f388b620e5da0ff06a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:58:40 UTC) from {username='slave'} at 127.0.0.1:48056
I20260927 03:58:40.118266 20977 tablet_service.cc:1514] Processing DeleteTablet for tablet f1b9c87258e74f388b620e5da0ff06a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:58:40 UTC) from {username='slave'} at 127.0.0.1:44822
I20260927 03:58:40.118294 21108 tablet_service.cc:1514] Processing DeleteTablet for tablet f1b9c87258e74f388b620e5da0ff06a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:58:40 UTC) from {username='slave'} at 127.0.0.1:37928
I20260927 03:58:40.118517 21212 tablet_replica.cc:330] T f1b9c87258e74f388b620e5da0ff06a5 P 234b4e84873d4de3addba45bff5cb20d: stopping tablet replica
I20260927 03:58:40.118618 21212 raft_consensus.cc:2229] T f1b9c87258e74f388b620e5da0ff06a5 P 234b4e84873d4de3addba45bff5cb20d [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:58:40.118705 21211 tablet_replica.cc:330] T f1b9c87258e74f388b620e5da0ff06a5 P 322e31085562442ea93349fc84078086: stopping tablet replica
I20260927 03:58:40.118722 21212 raft_consensus.cc:2258] T f1b9c87258e74f388b620e5da0ff06a5 P 234b4e84873d4de3addba45bff5cb20d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:58:40.118806 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 20784
I20260927 03:58:40.119385 21211 raft_consensus.cc:2229] T f1b9c87258e74f388b620e5da0ff06a5 P 322e31085562442ea93349fc84078086 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:58:40.119580 21211 raft_consensus.cc:2258] T f1b9c87258e74f388b620e5da0ff06a5 P 322e31085562442ea93349fc84078086 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:58:40.120183 21211 ts_tablet_manager.cc:1905] T f1b9c87258e74f388b620e5da0ff06a5 P 322e31085562442ea93349fc84078086: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:58:40.121917 21211 ts_tablet_manager.cc:1918] T f1b9c87258e74f388b620e5da0ff06a5 P 322e31085562442ea93349fc84078086: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:58:40.122005 21211 log.cc:1196] T f1b9c87258e74f388b620e5da0ff06a5 P 322e31085562442ea93349fc84078086: Deleting WAL directory at /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/ts-1/wal/wals/f1b9c87258e74f388b620e5da0ff06a5
I20260927 03:58:40.122267 21211 ts_tablet_manager.cc:1939] T f1b9c87258e74f388b620e5da0ff06a5 P 322e31085562442ea93349fc84078086: Deleting consensus metadata
I20260927 03:58:40.122864 20709 catalog_manager.cc:4841] TS 322e31085562442ea93349fc84078086 (127.15.185.66:37749): tablet f1b9c87258e74f388b620e5da0ff06a5 (table test-table [id=88f162ae699048ea8368f8319bcb2567]) successfully deleted
W20260927 03:58:40.124454 20708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.185.65:38547 (error 108)
I20260927 03:58:40.124989 21182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:40.124841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.65:38547 (local address 127.0.0.1:48068)
0927 03:58:40.124906 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:40.124948 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.65:38547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 03:58:40.125057 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 20915
I20260927 03:58:40.125303 21212 ts_tablet_manager.cc:1905] T f1b9c87258e74f388b620e5da0ff06a5 P 234b4e84873d4de3addba45bff5cb20d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:58:40.126844 21212 ts_tablet_manager.cc:1918] T f1b9c87258e74f388b620e5da0ff06a5 P 234b4e84873d4de3addba45bff5cb20d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:58:40.126916 21212 log.cc:1196] T f1b9c87258e74f388b620e5da0ff06a5 P 234b4e84873d4de3addba45bff5cb20d: Deleting WAL directory at /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481498018407-16101-0/minicluster-data/ts-2/wal/wals/f1b9c87258e74f388b620e5da0ff06a5
I20260927 03:58:40.127209 21212 ts_tablet_manager.cc:1939] T f1b9c87258e74f388b620e5da0ff06a5 P 234b4e84873d4de3addba45bff5cb20d: Deleting consensus metadata
W20260927 03:58:40.128371 20708 catalog_manager.cc:4584] TS 7662bf33e835431398984502168ba1e4 (127.15.185.65:38547): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f1b9c87258e74f388b620e5da0ff06a5: Network error: Client connection negotiation failed: client connection to 127.15.185.65:38547: connect: Connection refused (error 111)
I20260927 03:58:40.128497 20708 catalog_manager.cc:4841] TS 234b4e84873d4de3addba45bff5cb20d (127.15.185.67:36897): tablet f1b9c87258e74f388b620e5da0ff06a5 (table test-table [id=88f162ae699048ea8368f8319bcb2567]) successfully deleted
I20260927 03:58:40.132699 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 21046
I20260927 03:58:40.136703 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 20693
2026-09-27T03:58:40Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (749 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260927 03:58:40.156199 16101 mini_oidc.cc:152] Starting JWKS server
I20260927 03:58:40.157282 16101 webserver.cc:460] Webserver started at https://127.0.0.1:41031/ using document root <none> and password file <none>
I20260927 03:58:40.157366 16101 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 03:58:40.157569 16101 webserver.cc:460] Webserver started at http://127.0.0.1:37263/ using document root <none> and password file <none>
2026-09-27T03:58:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:58:40Z Disabled control of system clock
I20260927 03:58:40.165378 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790481498018407-16101-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790481498018407-16101-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790481498018407-16101-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:36983
--webserver_interface=127.15.185.126
--webserver_port=0
--builtin_ntp_servers=127.15.185.84:36759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.185.126:36983
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41031/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790481498018407-16101-0/testchainca.cert with env {}
W20260927 03:58:40.245326 21219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:40.245530 21219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:40.245553 21219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:40.246716 21219 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:58:40.246755 21219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:40.246768 21219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:40.246780 21219 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:58:40.246793 21219 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 03:58:40.246845 21219 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 03:58:40.246862 21219 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41031/jwks.json
I20260927 03:58:40.248394 21219 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:36759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790481498018407-16101-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790481498018407-16101-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.185.126:36983
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:36983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41031/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.185.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790481498018407-16101-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790481498018407-16101-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:40.248651 21219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:40.248867 21219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:40.253674 21225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:40.253707 21224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:40.253938 21227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:40.253993 21219 server_base.cc:1034] running on GCE node
I20260927 03:58:40.254231 21219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:40.254477 21219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:40.255614 21219 hybrid_clock.cc:648] HybridClock initialized: now 1790481520255604 us; error 33 us; skew 500 ppm
I20260927 03:58:40.256867 21219 webserver.cc:460] Webserver started at http://127.15.185.126:32815/ using document root <none> and password file <none>
I20260927 03:58:40.257110 21219 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:40.257157 21219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:40.257298 21219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:40.258325 21219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790481498018407-16101-0/minicluster-data/master-0/data/instance:
uuid: "69ccb3775ad546f0a988540b955011e7"
format_stamp: "Formatted at 2026-09-27 03:58:40 on dist-test-slave-glgz"
server_key: "5faf2abc32dc31836aa7aafe8bd54e74"
server_key_iv: "31540c9bfbef73a38fd6a49ce8077e18"
server_key_version: "encryptionkey@0"
I20260927 03:58:40.258697 21219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790481498018407-16101-0/minicluster-data/master-0/wal/instance:
uuid: "69ccb3775ad546f0a988540b955011e7"
format_stamp: "Formatted at 2026-09-27 03:58:40 on dist-test-slave-glgz"
server_key: "5faf2abc32dc31836aa7aafe8bd54e74"
server_key_iv: "31540c9bfbef73a38fd6a49ce8077e18"
server_key_version: "encryptionkey@0"
I20260927 03:58:40.260048 21219 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:58:40.260792 21233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:40.260973 21219 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:58:40.261085 21219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790481498018407-16101-0/minicluster-data/master-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790481498018407-16101-0/minicluster-data/master-0/wal
uuid: "69ccb3775ad546f0a988540b955011e7"
format_stamp: "Formatted at 2026-09-27 03:58:40 on dist-test-slave-glgz"
server_key: "5faf2abc32dc31836aa7aafe8bd54e74"
server_key_iv: "31540c9bfbef73a38fd6a49ce8077e18"
server_key_version: "encryptionkey@0"
I20260927 03:58:40.261159 21219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790481498018407-16101-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790481498018407-16101-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790481498018407-16101-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:40.281064 21219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:40.281410 21219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:40.281563 21219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:40.285658 21219 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.126:36983
I20260927 03:58:40.285686 21285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.126:36983 every 8 connection(s)
I20260927 03:58:40.286088 21219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
I20260927 03:58:40.286990 21286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:40.289258 21286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69ccb3775ad546f0a988540b955011e7: Bootstrap starting.
I20260927 03:58:40.289623 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 21219
I20260927 03:58:40.289733 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790481498018407-16101-0/minicluster-data/master-0/wal/instance
I20260927 03:58:40.289844 21286 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69ccb3775ad546f0a988540b955011e7: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:40.289948 16101 external_mini_cluster.cc:1428] Setting key 7585009618f61ba9408d80d4a1ff645e
I20260927 03:58:40.290197 21286 log.cc:824] T 00000000000000000000000000000000 P 69ccb3775ad546f0a988540b955011e7: Log is configured to *not* fsync() on all Append() calls
I20260927 03:58:40.290891 21286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69ccb3775ad546f0a988540b955011e7: No bootstrap required, opened a new log
I20260927 03:58:40.292693 21286 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69ccb3775ad546f0a988540b955011e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69ccb3775ad546f0a988540b955011e7" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 36983 } }
I20260927 03:58:40.292851 21286 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69ccb3775ad546f0a988540b955011e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:40.292898 21286 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69ccb3775ad546f0a988540b955011e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ccb3775ad546f0a988540b955011e7, State: Initialized, Role: FOLLOWER
I20260927 03:58:40.293077 21286 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69ccb3775ad546f0a988540b955011e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69ccb3775ad546f0a988540b955011e7" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 36983 } }
I20260927 03:58:40.293174 21286 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69ccb3775ad546f0a988540b955011e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:58:40.293216 21286 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69ccb3775ad546f0a988540b955011e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:58:40.293263 21286 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69ccb3775ad546f0a988540b955011e7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:40.293783 21286 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69ccb3775ad546f0a988540b955011e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69ccb3775ad546f0a988540b955011e7" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 36983 } }
I20260927 03:58:40.293902 21286 leader_election.cc:302] T 00000000000000000000000000000000 P 69ccb3775ad546f0a988540b955011e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69ccb3775ad546f0a988540b955011e7; no voters:
I20260927 03:58:40.294088 21286 leader_election.cc:288] T 00000000000000000000000000000000 P 69ccb3775ad546f0a988540b955011e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:58:40.294112 21291 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69ccb3775ad546f0a988540b955011e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:58:40.294370 21286 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69ccb3775ad546f0a988540b955011e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:58:40.294566 21291 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69ccb3775ad546f0a988540b955011e7 [term 1 LEADER]: Becoming Leader. State: Replica: 69ccb3775ad546f0a988540b955011e7, State: Running, Role: LEADER
I20260927 03:58:40.294667 21291 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69ccb3775ad546f0a988540b955011e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69ccb3775ad546f0a988540b955011e7" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 36983 } }
I20260927 03:58:40.295130 21292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69ccb3775ad546f0a988540b955011e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69ccb3775ad546f0a988540b955011e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69ccb3775ad546f0a988540b955011e7" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 36983 } } }
I20260927 03:58:40.295135 21293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69ccb3775ad546f0a988540b955011e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69ccb3775ad546f0a988540b955011e7. Latest consensus state: current_term: 1 leader_uuid: "69ccb3775ad546f0a988540b955011e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69ccb3775ad546f0a988540b955011e7" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 36983 } } }
I20260927 03:58:40.295231 21292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69ccb3775ad546f0a988540b955011e7 [sys.catalog]: This master's current role is: LEADER
I20260927 03:58:40.295250 21293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69ccb3775ad546f0a988540b955011e7 [sys.catalog]: This master's current role is: LEADER
I20260927 03:58:40.295459 21300 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:58:40.295974 21300 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:58:40.296051 21290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:40.290450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39832 (local address 127.15.185.126:36983)
0927 03:58:40.290665 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:40.290674 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:40.290706 (+ 32us) server_negotiation.cc:407] Connection header received
0927 03:58:40.291156 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:40.291163 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:40.291223 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:40.291364 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:40.291588 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:40.292440 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:40.293396 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:40.293805 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:40.293974 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:40.294700 (+ 726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:40.294704 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:40.294704 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:58:40.294950 (+ 246us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:40.294995 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:58:40.295010 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:40.295673 (+ 663us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:40.295707 (+ 34us) server_negotiation.cc:299] Negotiation successful
0927 03:58:40.295792 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":79,"threads_started":1}
I20260927 03:58:40.296886 16101 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:58:40.297551 21300 catalog_manager.cc:1347] Generated new cluster ID: f96a94eb84394b608c72d41d8120e2a0
I20260927 03:58:40.297607 21300 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:58:40.306915 21300 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:58:40.307498 21300 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:58:40.309804 21300 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69ccb3775ad546f0a988540b955011e7: Generated new TSK 0
I20260927 03:58:40.309986 21300 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 03:58:40.342639 21316 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260927 03:58:40.342748 21316 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260927 03:58:40.343394 21315 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:58:40.320752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:36983 (local address 127.0.0.1:39840)
0927 03:58:40.320944 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:40.320956 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:40.321029 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:40.321192 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:40.321192 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:40.321194 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0927 03:58:40.321260 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:40.321262 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:40.321894 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:40.321895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:40.322863 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:40.322865 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:40.323290 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:40.323291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:40.323361 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:40.323383 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 03:58:40.342910 (+ 19527us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41031/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 03:58:40.343022 (+ 112us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.15.185.126:36983: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41031/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":104,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.15.185.126:36983: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41031/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260927 03:58:40.343706 21290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:40.320859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39840 (local address 127.15.185.126:36983)
0927 03:58:40.320904 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:40.320907 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:40.321022 (+ 115us) server_negotiation.cc:407] Connection header received
0927 03:58:40.321120 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:40.321120 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:40.321130 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:40.321186 (+ 56us) server_negotiation.cc:226] Negotiated authn=JWT
0927 03:58:40.321322 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:40.321804 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:40.322967 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:40.323208 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:40.323261 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:40.323498 (+ 237us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 03:58:40.342761 (+ 19263us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41031/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 03:58:40.343640 (+ 879us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39840: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41031/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 03:58:40.343746 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 21219
W20260927 03:58:40.343763 21290 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39840: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41031/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
2026-09-27T03:58:40Z chronyd exiting
I20260927 03:58:40.379786 16101 test_util.cc:175] -----------------------------------------------
I20260927 03:58:40.379879 16101 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790481498018407-16101-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (224 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260927 03:58:40.380610 16101 mini_oidc.cc:152] Starting JWKS server
I20260927 03:58:40.381325 16101 webserver.cc:460] Webserver started at https://127.0.0.1:35561/ using document root <none> and password file <none>
I20260927 03:58:40.381371 16101 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 03:58:40.381515 16101 webserver.cc:460] Webserver started at http://127.0.0.1:36435/ using document root <none> and password file <none>
2026-09-27T03:58:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:58:40Z Disabled control of system clock
I20260927 03:58:40.389561 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790481498018407-16101-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790481498018407-16101-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790481498018407-16101-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:35271
--webserver_interface=127.15.185.126
--webserver_port=0
--builtin_ntp_servers=127.15.185.84:39325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.185.126:35271
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35561/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790481498018407-16101-0/testchainca.cert with env {}
W20260927 03:58:40.470613 21322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:40.470804 21322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:40.470825 21322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:40.471925 21322 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:58:40.471961 21322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:40.471974 21322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:40.471987 21322 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:58:40.471999 21322 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 03:58:40.472011 21322 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 03:58:40.472023 21322 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35561/jwks.json
I20260927 03:58:40.473418 21322 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:39325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790481498018407-16101-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790481498018407-16101-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.185.126:35271
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:35271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35561/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.185.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790481498018407-16101-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790481498018407-16101-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:40.473600 21322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:40.473744 21322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:40.478531 21328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:40.478504 21330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:40.478505 21327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:40.478824 21322 server_base.cc:1034] running on GCE node
I20260927 03:58:40.479110 21322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:40.479378 21322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:40.480566 21322 hybrid_clock.cc:648] HybridClock initialized: now 1790481520480551 us; error 28 us; skew 500 ppm
I20260927 03:58:40.482028 21322 webserver.cc:460] Webserver started at http://127.15.185.126:35691/ using document root <none> and password file <none>
I20260927 03:58:40.482276 21322 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:40.482323 21322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:40.482448 21322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:40.483559 21322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790481498018407-16101-0/minicluster-data/master-0/data/instance:
uuid: "5a5689f2880f44d2b4502e1b6c2b9d8b"
format_stamp: "Formatted at 2026-09-27 03:58:40 on dist-test-slave-glgz"
server_key: "d4d413228f79985f5b6b55f44a39c748"
server_key_iv: "20b022ea992d4f756fa13fc2ce3e8913"
server_key_version: "encryptionkey@0"
I20260927 03:58:40.483907 21322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790481498018407-16101-0/minicluster-data/master-0/wal/instance:
uuid: "5a5689f2880f44d2b4502e1b6c2b9d8b"
format_stamp: "Formatted at 2026-09-27 03:58:40 on dist-test-slave-glgz"
server_key: "d4d413228f79985f5b6b55f44a39c748"
server_key_iv: "20b022ea992d4f756fa13fc2ce3e8913"
server_key_version: "encryptionkey@0"
I20260927 03:58:40.485193 21322 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:58:40.485911 21336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:40.486080 21322 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:58:40.486181 21322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790481498018407-16101-0/minicluster-data/master-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790481498018407-16101-0/minicluster-data/master-0/wal
uuid: "5a5689f2880f44d2b4502e1b6c2b9d8b"
format_stamp: "Formatted at 2026-09-27 03:58:40 on dist-test-slave-glgz"
server_key: "d4d413228f79985f5b6b55f44a39c748"
server_key_iv: "20b022ea992d4f756fa13fc2ce3e8913"
server_key_version: "encryptionkey@0"
I20260927 03:58:40.486253 21322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790481498018407-16101-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790481498018407-16101-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790481498018407-16101-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:40.504146 21322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:40.504462 21322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:40.504616 21322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:40.508520 21322 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.126:35271
I20260927 03:58:40.508558 21388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.126:35271 every 8 connection(s)
I20260927 03:58:40.508903 21322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
I20260927 03:58:40.509577 21389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:40.512321 21389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a5689f2880f44d2b4502e1b6c2b9d8b: Bootstrap starting.
I20260927 03:58:40.512840 21389 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a5689f2880f44d2b4502e1b6c2b9d8b: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:40.513070 21389 log.cc:824] T 00000000000000000000000000000000 P 5a5689f2880f44d2b4502e1b6c2b9d8b: Log is configured to *not* fsync() on all Append() calls
I20260927 03:58:40.513649 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 21322
I20260927 03:58:40.513751 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790481498018407-16101-0/minicluster-data/master-0/wal/instance
I20260927 03:58:40.513789 21389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a5689f2880f44d2b4502e1b6c2b9d8b: No bootstrap required, opened a new log
I20260927 03:58:40.513952 16101 external_mini_cluster.cc:1428] Setting key fefe3908a553b27571417fde6013ed62
I20260927 03:58:40.516757 21389 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a5689f2880f44d2b4502e1b6c2b9d8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a5689f2880f44d2b4502e1b6c2b9d8b" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 35271 } }
I20260927 03:58:40.517001 21389 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a5689f2880f44d2b4502e1b6c2b9d8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:40.517042 21389 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a5689f2880f44d2b4502e1b6c2b9d8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a5689f2880f44d2b4502e1b6c2b9d8b, State: Initialized, Role: FOLLOWER
I20260927 03:58:40.517180 21389 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a5689f2880f44d2b4502e1b6c2b9d8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a5689f2880f44d2b4502e1b6c2b9d8b" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 35271 } }
I20260927 03:58:40.517252 21389 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a5689f2880f44d2b4502e1b6c2b9d8b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:58:40.517280 21389 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a5689f2880f44d2b4502e1b6c2b9d8b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:58:40.517319 21389 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a5689f2880f44d2b4502e1b6c2b9d8b [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:40.517974 21389 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a5689f2880f44d2b4502e1b6c2b9d8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a5689f2880f44d2b4502e1b6c2b9d8b" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 35271 } }
I20260927 03:58:40.518064 21389 leader_election.cc:302] T 00000000000000000000000000000000 P 5a5689f2880f44d2b4502e1b6c2b9d8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a5689f2880f44d2b4502e1b6c2b9d8b; no voters:
I20260927 03:58:40.518282 21389 leader_election.cc:288] T 00000000000000000000000000000000 P 5a5689f2880f44d2b4502e1b6c2b9d8b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:58:40.518301 21394 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a5689f2880f44d2b4502e1b6c2b9d8b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:58:40.518437 21394 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a5689f2880f44d2b4502e1b6c2b9d8b [term 1 LEADER]: Becoming Leader. State: Replica: 5a5689f2880f44d2b4502e1b6c2b9d8b, State: Running, Role: LEADER
I20260927 03:58:40.518519 21394 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a5689f2880f44d2b4502e1b6c2b9d8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a5689f2880f44d2b4502e1b6c2b9d8b" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 35271 } }
I20260927 03:58:40.518548 21389 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a5689f2880f44d2b4502e1b6c2b9d8b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:58:40.519726 21393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:40.514471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60170 (local address 127.15.185.126:35271)
0927 03:58:40.514669 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:40.514676 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:40.514702 (+ 26us) server_negotiation.cc:407] Connection header received
0927 03:58:40.515247 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:40.515254 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:40.515315 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:40.515445 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:40.515659 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:40.516562 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:40.517647 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:40.518157 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:40.518287 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:40.519113 (+ 826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:40.519117 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:40.519117 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:58:40.519291 (+ 174us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:40.519323 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:58:40.519333 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:40.519411 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:40.519484 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 03:58:40.519567 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":79,"threads_started":1}
I20260927 03:58:40.520758 21395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a5689f2880f44d2b4502e1b6c2b9d8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a5689f2880f44d2b4502e1b6c2b9d8b. Latest consensus state: current_term: 1 leader_uuid: "5a5689f2880f44d2b4502e1b6c2b9d8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a5689f2880f44d2b4502e1b6c2b9d8b" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 35271 } } }
I20260927 03:58:40.520925 16101 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:58:40.520776 21396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a5689f2880f44d2b4502e1b6c2b9d8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a5689f2880f44d2b4502e1b6c2b9d8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a5689f2880f44d2b4502e1b6c2b9d8b" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 35271 } } }
I20260927 03:58:40.520978 21396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a5689f2880f44d2b4502e1b6c2b9d8b [sys.catalog]: This master's current role is: LEADER
I20260927 03:58:40.520866 21395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a5689f2880f44d2b4502e1b6c2b9d8b [sys.catalog]: This master's current role is: LEADER
W20260927 03:58:40.520918 21408 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5a5689f2880f44d2b4502e1b6c2b9d8b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:58:40.521625 21408 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:58:40.521678 21411 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:58:40.521859 21411 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:58:40.523309 21411 catalog_manager.cc:1347] Generated new cluster ID: 1628498998884a9089493f88a319633c
I20260927 03:58:40.523360 21411 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:58:40.538180 21411 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:58:40.538710 21411 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:58:40.542265 21411 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a5689f2880f44d2b4502e1b6c2b9d8b: Generated new TSK 0
I20260927 03:58:40.542434 21411 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 03:58:40.565810 21419 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260927 03:58:40.565899 21419 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 03:58:40.566247 21393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:40.551771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60186 (local address 127.15.185.126:35271)
0927 03:58:40.551816 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:40.551818 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:40.551927 (+ 109us) server_negotiation.cc:407] Connection header received
0927 03:58:40.551994 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:40.551994 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:40.552006 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:40.552066 (+ 60us) server_negotiation.cc:226] Negotiated authn=JWT
0927 03:58:40.552231 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:40.552732 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:40.553923 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:40.554122 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:40.554185 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:40.554359 (+ 174us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 03:58:40.566073 (+ 11714us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35561/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 03:58:40.566194 (+ 121us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60186: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35561/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":24}
W20260927 03:58:40.566332 21393 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60186: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35561/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260927 03:58:40.566438 21418 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:58:40.551671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:35271 (local address 127.0.0.1:60186)
0927 03:58:40.551845 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:40.551864 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:40.551927 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:40.552080 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:40.552081 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:40.552083 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0927 03:58:40.552143 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:40.552146 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:40.552815 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:40.552816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:40.553782 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:40.553785 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:40.554202 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:40.554203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:40.554264 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:40.554269 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 03:58:40.566188 (+ 11919us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35561/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 03:58:40.566332 (+ 144us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.15.185.126:35271: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35561/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":99,"threads_started":1}
I20260927 03:58:40.566839 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 21322
2026-09-27T03:58:40Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (224 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260927 03:58:40.604776 16101 mini_oidc.cc:152] Starting JWKS server
I20260927 03:58:40.605523 16101 webserver.cc:460] Webserver started at https://127.0.0.1:34279/ using document root <none> and password file <none>
I20260927 03:58:40.605568 16101 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 03:58:40.605762 16101 webserver.cc:460] Webserver started at http://127.0.0.1:45379/ using document root <none> and password file <none>
2026-09-27T03:58:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:58:40Z Disabled control of system clock
I20260927 03:58:40.613137 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790481498018407-16101-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790481498018407-16101-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790481498018407-16101-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:34133
--webserver_interface=127.15.185.126
--webserver_port=0
--builtin_ntp_servers=127.15.185.84:38569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.185.126:34133
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34279/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 03:58:40.692170 21425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:40.692435 21425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:40.692472 21425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:40.693671 21425 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:58:40.693742 21425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:40.693763 21425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:40.693784 21425 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:58:40.693815 21425 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 03:58:40.693862 21425 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 03:58:40.693892 21425 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34279/jwks.json
I20260927 03:58:40.695252 21425 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:38569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790481498018407-16101-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790481498018407-16101-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.185.126:34133
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:34133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34279/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790481498018407-16101-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:40.695528 21425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:40.695819 21425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:40.700639 21430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:40.700639 21431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:40.700655 21433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:40.700932 21425 server_base.cc:1034] running on GCE node
I20260927 03:58:40.701162 21425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:40.701436 21425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:40.702574 21425 hybrid_clock.cc:648] HybridClock initialized: now 1790481520702552 us; error 37 us; skew 500 ppm
I20260927 03:58:40.703809 21425 webserver.cc:460] Webserver started at http://127.15.185.126:35023/ using document root <none> and password file <none>
I20260927 03:58:40.704062 21425 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:40.704108 21425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:40.704241 21425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:40.705366 21425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790481498018407-16101-0/minicluster-data/master-0/data/instance:
uuid: "c486ae90e4b549749dff14d252973ccf"
format_stamp: "Formatted at 2026-09-27 03:58:40 on dist-test-slave-glgz"
server_key: "8a856eb25fb516d852068c20c798b783"
server_key_iv: "c03ebcd8c784f86f2363570f2d46ecbc"
server_key_version: "encryptionkey@0"
I20260927 03:58:40.705721 21425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790481498018407-16101-0/minicluster-data/master-0/wal/instance:
uuid: "c486ae90e4b549749dff14d252973ccf"
format_stamp: "Formatted at 2026-09-27 03:58:40 on dist-test-slave-glgz"
server_key: "8a856eb25fb516d852068c20c798b783"
server_key_iv: "c03ebcd8c784f86f2363570f2d46ecbc"
server_key_version: "encryptionkey@0"
I20260927 03:58:40.707113 21425 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:58:40.707849 21439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:40.708038 21425 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:58:40.708128 21425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790481498018407-16101-0/minicluster-data/master-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790481498018407-16101-0/minicluster-data/master-0/wal
uuid: "c486ae90e4b549749dff14d252973ccf"
format_stamp: "Formatted at 2026-09-27 03:58:40 on dist-test-slave-glgz"
server_key: "8a856eb25fb516d852068c20c798b783"
server_key_iv: "c03ebcd8c784f86f2363570f2d46ecbc"
server_key_version: "encryptionkey@0"
I20260927 03:58:40.708200 21425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790481498018407-16101-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790481498018407-16101-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790481498018407-16101-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:40.722752 21425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:40.723086 21425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:40.723243 21425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:40.727175 21425 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.126:34133
I20260927 03:58:40.727206 21491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.126:34133 every 8 connection(s)
I20260927 03:58:40.727545 21425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
I20260927 03:58:40.728142 21492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:40.730868 21492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c486ae90e4b549749dff14d252973ccf: Bootstrap starting.
I20260927 03:58:40.731550 21492 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c486ae90e4b549749dff14d252973ccf: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:40.731840 21492 log.cc:824] T 00000000000000000000000000000000 P c486ae90e4b549749dff14d252973ccf: Log is configured to *not* fsync() on all Append() calls
I20260927 03:58:40.732479 21492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c486ae90e4b549749dff14d252973ccf: No bootstrap required, opened a new log
I20260927 03:58:40.734325 21492 raft_consensus.cc:348] T 00000000000000000000000000000000 P c486ae90e4b549749dff14d252973ccf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c486ae90e4b549749dff14d252973ccf" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 34133 } }
I20260927 03:58:40.734465 21492 raft_consensus.cc:374] T 00000000000000000000000000000000 P c486ae90e4b549749dff14d252973ccf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:40.734488 21492 raft_consensus.cc:729] T 00000000000000000000000000000000 P c486ae90e4b549749dff14d252973ccf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c486ae90e4b549749dff14d252973ccf, State: Initialized, Role: FOLLOWER
I20260927 03:58:40.734666 21492 consensus_queue.cc:260] T 00000000000000000000000000000000 P c486ae90e4b549749dff14d252973ccf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c486ae90e4b549749dff14d252973ccf" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 34133 } }
I20260927 03:58:40.734730 21492 raft_consensus.cc:388] T 00000000000000000000000000000000 P c486ae90e4b549749dff14d252973ccf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:58:40.734751 21492 raft_consensus.cc:482] T 00000000000000000000000000000000 P c486ae90e4b549749dff14d252973ccf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:58:40.734824 21492 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c486ae90e4b549749dff14d252973ccf [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:40.735384 21492 raft_consensus.cc:504] T 00000000000000000000000000000000 P c486ae90e4b549749dff14d252973ccf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c486ae90e4b549749dff14d252973ccf" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 34133 } }
I20260927 03:58:40.735495 21492 leader_election.cc:302] T 00000000000000000000000000000000 P c486ae90e4b549749dff14d252973ccf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c486ae90e4b549749dff14d252973ccf; no voters:
I20260927 03:58:40.735664 21492 leader_election.cc:288] T 00000000000000000000000000000000 P c486ae90e4b549749dff14d252973ccf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:58:40.735693 21495 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c486ae90e4b549749dff14d252973ccf [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:58:40.735854 21495 raft_consensus.cc:686] T 00000000000000000000000000000000 P c486ae90e4b549749dff14d252973ccf [term 1 LEADER]: Becoming Leader. State: Replica: c486ae90e4b549749dff14d252973ccf, State: Running, Role: LEADER
I20260927 03:58:40.735927 21492 sys_catalog.cc:564] T 00000000000000000000000000000000 P c486ae90e4b549749dff14d252973ccf [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:58:40.735948 21495 consensus_queue.cc:237] T 00000000000000000000000000000000 P c486ae90e4b549749dff14d252973ccf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c486ae90e4b549749dff14d252973ccf" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 34133 } }
I20260927 03:58:40.736335 21496 sys_catalog.cc:455] T 00000000000000000000000000000000 P c486ae90e4b549749dff14d252973ccf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c486ae90e4b549749dff14d252973ccf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c486ae90e4b549749dff14d252973ccf" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 34133 } } }
I20260927 03:58:40.736423 21496 sys_catalog.cc:458] T 00000000000000000000000000000000 P c486ae90e4b549749dff14d252973ccf [sys.catalog]: This master's current role is: LEADER
I20260927 03:58:40.736649 21497 sys_catalog.cc:455] T 00000000000000000000000000000000 P c486ae90e4b549749dff14d252973ccf [sys.catalog]: SysCatalogTable state changed. Reason: New leader c486ae90e4b549749dff14d252973ccf. Latest consensus state: current_term: 1 leader_uuid: "c486ae90e4b549749dff14d252973ccf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c486ae90e4b549749dff14d252973ccf" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 34133 } } }
I20260927 03:58:40.736725 21497 sys_catalog.cc:458] T 00000000000000000000000000000000 P c486ae90e4b549749dff14d252973ccf [sys.catalog]: This master's current role is: LEADER
I20260927 03:58:40.736738 21502 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:58:40.737277 21502 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:58:40.737377 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 21425
I20260927 03:58:40.737481 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790481498018407-16101-0/minicluster-data/master-0/wal/instance
I20260927 03:58:40.737676 16101 external_mini_cluster.cc:1428] Setting key a0af4498759f3cf2782ca60aedb29da9
I20260927 03:58:40.739017 21502 catalog_manager.cc:1347] Generated new cluster ID: 9908ebac668a4322a07025c0e24a9d41
I20260927 03:58:40.739090 21502 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:58:40.742802 21514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:40.738189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45896 (local address 127.15.185.126:34133)
0927 03:58:40.738377 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:40.738387 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:40.738462 (+ 75us) server_negotiation.cc:407] Connection header received
0927 03:58:40.738705 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:40.738711 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:40.738763 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:40.738907 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:40.739132 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:40.740018 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:40.740800 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:40.741263 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:40.741388 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:40.742118 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:40.742122 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:40.742125 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:58:40.742291 (+ 166us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:40.742322 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:58:40.742332 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:40.742406 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:40.742489 (+ 83us) server_negotiation.cc:299] Negotiation successful
0927 03:58:40.742575 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":87,"threads_started":1}
I20260927 03:58:40.743743 16101 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:58:40.747465 21502 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:58:40.747949 21502 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:58:40.750615 21502 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c486ae90e4b549749dff14d252973ccf: Generated new TSK 0
I20260927 03:58:40.750792 21502 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 03:58:40.763774 21521 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:58:40.759229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34133 (local address 127.0.0.1:45906)
0927 03:58:40.759395 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:40.759405 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:40.759467 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:40.759613 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:40.759614 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:40.759615 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0927 03:58:40.759672 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:40.759673 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:40.760592 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:40.760593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:40.761263 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:40.761267 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:40.761827 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:40.761828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:40.761925 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:40.761932 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 03:58:40.763599 (+ 1667us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34279/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 03:58:40.763674 (+ 75us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.15.185.126:34133: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34279/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":103,"threads_started":1}
I20260927 03:58:40.764133 21514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:40.759319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45906 (local address 127.15.185.126:34133)
0927 03:58:40.759360 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:40.759362 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:40.759471 (+ 109us) server_negotiation.cc:407] Connection header received
0927 03:58:40.759531 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:40.759532 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:40.759544 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:40.759617 (+ 73us) server_negotiation.cc:226] Negotiated authn=JWT
0927 03:58:40.759723 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:40.760498 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:40.761354 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:40.761709 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:40.761771 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:40.762043 (+ 272us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 03:58:40.763481 (+ 1438us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34279/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 03:58:40.764060 (+ 579us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45906: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34279/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":19}
W20260927 03:58:40.764174 21514 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45906: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34279/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260927 03:58:40.766337 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 21425
E20260927 03:58:40.777697 21522 webserver.cc:533] Webserver: sslize failed:
E20260927 03:58:40.777793 21522 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-27T03:58:40Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (200 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-27T03:58:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:58:40Z Disabled control of system clock
I20260927 03:58:40.812232 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790481498018407-16101-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790481498018407-16101-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790481498018407-16101-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:41385
--webserver_interface=127.15.185.126
--webserver_port=0
--builtin_ntp_servers=127.15.185.84:34011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.185.126:41385
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790481498018407-16101-0/testchainca.cert with env {}
W20260927 03:58:40.891633 21526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:40.891836 21526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:40.891860 21526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:40.892971 21526 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:58:40.893009 21526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:40.893023 21526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:40.893034 21526 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:58:40.893046 21526 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 03:58:40.893059 21526 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 03:58:40.893070 21526 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260927 03:58:40.894526 21526 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:34011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790481498018407-16101-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790481498018407-16101-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.185.126:41385
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:41385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.185.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790481498018407-16101-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790481498018407-16101-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:40.894729 21526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:40.894892 21526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:40.900051 21532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:40.900017 21531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:40.900202 21526 server_base.cc:1034] running on GCE node
W20260927 03:58:40.900034 21534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:40.900514 21526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:40.900820 21526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:40.902002 21526 hybrid_clock.cc:648] HybridClock initialized: now 1790481520901987 us; error 41 us; skew 500 ppm
I20260927 03:58:40.903380 21526 webserver.cc:460] Webserver started at http://127.15.185.126:41037/ using document root <none> and password file <none>
I20260927 03:58:40.903671 21526 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:40.903743 21526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:40.903892 21526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:40.905004 21526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790481498018407-16101-0/minicluster-data/master-0/data/instance:
uuid: "25d9f6acc04d4684a92ab68e4aeb4376"
format_stamp: "Formatted at 2026-09-27 03:58:40 on dist-test-slave-glgz"
server_key: "aa9874ac0ce8d8a35374001b099ecaee"
server_key_iv: "93b9ab1b1d978ecb1b4d3be27ea95004"
server_key_version: "encryptionkey@0"
I20260927 03:58:40.905352 21526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790481498018407-16101-0/minicluster-data/master-0/wal/instance:
uuid: "25d9f6acc04d4684a92ab68e4aeb4376"
format_stamp: "Formatted at 2026-09-27 03:58:40 on dist-test-slave-glgz"
server_key: "aa9874ac0ce8d8a35374001b099ecaee"
server_key_iv: "93b9ab1b1d978ecb1b4d3be27ea95004"
server_key_version: "encryptionkey@0"
I20260927 03:58:40.906711 21526 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:58:40.907485 21540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:40.907656 21526 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:40.907757 21526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790481498018407-16101-0/minicluster-data/master-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790481498018407-16101-0/minicluster-data/master-0/wal
uuid: "25d9f6acc04d4684a92ab68e4aeb4376"
format_stamp: "Formatted at 2026-09-27 03:58:40 on dist-test-slave-glgz"
server_key: "aa9874ac0ce8d8a35374001b099ecaee"
server_key_iv: "93b9ab1b1d978ecb1b4d3be27ea95004"
server_key_version: "encryptionkey@0"
I20260927 03:58:40.907828 21526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790481498018407-16101-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790481498018407-16101-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790481498018407-16101-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:40.930899 21526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:40.931303 21526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:40.931461 21526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:40.935683 21526 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.126:41385
I20260927 03:58:40.935714 21592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.126:41385 every 8 connection(s)
I20260927 03:58:40.936077 21526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
I20260927 03:58:40.936439 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 21526
I20260927 03:58:40.936565 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790481498018407-16101-0/minicluster-data/master-0/wal/instance
I20260927 03:58:40.936756 16101 external_mini_cluster.cc:1428] Setting key 80b25e8626c2f289795e2a3123b4e0c4
I20260927 03:58:40.937242 21593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:40.940675 21593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25d9f6acc04d4684a92ab68e4aeb4376: Bootstrap starting.
I20260927 03:58:40.941355 21593 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25d9f6acc04d4684a92ab68e4aeb4376: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:40.941653 21593 log.cc:824] T 00000000000000000000000000000000 P 25d9f6acc04d4684a92ab68e4aeb4376: Log is configured to *not* fsync() on all Append() calls
I20260927 03:58:40.942489 21593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25d9f6acc04d4684a92ab68e4aeb4376: No bootstrap required, opened a new log
I20260927 03:58:40.943248 21596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:40.937374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36046 (local address 127.15.185.126:41385)
0927 03:58:40.937614 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:40.937624 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:40.937657 (+ 33us) server_negotiation.cc:407] Connection header received
0927 03:58:40.938383 (+ 726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:40.938393 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:40.938481 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:40.938641 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:40.938797 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:40.940201 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:40.941012 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:40.941644 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:40.941774 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:40.942529 (+ 755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:40.942533 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:40.942534 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:58:40.942746 (+ 212us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:40.942786 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:58:40.942800 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:40.942869 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:40.942932 (+ 63us) server_negotiation.cc:299] Negotiation successful
0927 03:58:40.943013 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":100,"threads_started":1}
I20260927 03:58:40.944273 16101 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 03:58:40.944694 21593 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25d9f6acc04d4684a92ab68e4aeb4376 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25d9f6acc04d4684a92ab68e4aeb4376" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 41385 } }
I20260927 03:58:40.944880 21593 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25d9f6acc04d4684a92ab68e4aeb4376 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:40.944912 21593 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25d9f6acc04d4684a92ab68e4aeb4376 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25d9f6acc04d4684a92ab68e4aeb4376, State: Initialized, Role: FOLLOWER
I20260927 03:58:40.945032 21593 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25d9f6acc04d4684a92ab68e4aeb4376 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25d9f6acc04d4684a92ab68e4aeb4376" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 41385 } }
I20260927 03:58:40.945127 21593 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25d9f6acc04d4684a92ab68e4aeb4376 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:58:40.945170 21593 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25d9f6acc04d4684a92ab68e4aeb4376 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:58:40.945220 21593 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25d9f6acc04d4684a92ab68e4aeb4376 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:40.945758 21593 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25d9f6acc04d4684a92ab68e4aeb4376 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25d9f6acc04d4684a92ab68e4aeb4376" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 41385 } }
I20260927 03:58:40.945873 21593 leader_election.cc:302] T 00000000000000000000000000000000 P 25d9f6acc04d4684a92ab68e4aeb4376 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25d9f6acc04d4684a92ab68e4aeb4376; no voters:
I20260927 03:58:40.946058 21593 leader_election.cc:288] T 00000000000000000000000000000000 P 25d9f6acc04d4684a92ab68e4aeb4376 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:58:40.946131 21598 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25d9f6acc04d4684a92ab68e4aeb4376 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:58:40.946301 21593 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25d9f6acc04d4684a92ab68e4aeb4376 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:58:40.946328 21598 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25d9f6acc04d4684a92ab68e4aeb4376 [term 1 LEADER]: Becoming Leader. State: Replica: 25d9f6acc04d4684a92ab68e4aeb4376, State: Running, Role: LEADER
I20260927 03:58:40.946410 21598 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25d9f6acc04d4684a92ab68e4aeb4376 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25d9f6acc04d4684a92ab68e4aeb4376" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 41385 } }
I20260927 03:58:40.946769 21599 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25d9f6acc04d4684a92ab68e4aeb4376 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25d9f6acc04d4684a92ab68e4aeb4376" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25d9f6acc04d4684a92ab68e4aeb4376" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 41385 } } }
I20260927 03:58:40.946887 21599 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25d9f6acc04d4684a92ab68e4aeb4376 [sys.catalog]: This master's current role is: LEADER
I20260927 03:58:40.946774 21600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25d9f6acc04d4684a92ab68e4aeb4376 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25d9f6acc04d4684a92ab68e4aeb4376. Latest consensus state: current_term: 1 leader_uuid: "25d9f6acc04d4684a92ab68e4aeb4376" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25d9f6acc04d4684a92ab68e4aeb4376" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 41385 } } }
I20260927 03:58:40.947134 21600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25d9f6acc04d4684a92ab68e4aeb4376 [sys.catalog]: This master's current role is: LEADER
I20260927 03:58:40.947381 21607 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:58:40.947822 21607 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:58:40.949342 21607 catalog_manager.cc:1347] Generated new cluster ID: 96578e52ee4a416aa3cebd7b6cea6991
I20260927 03:58:40.949384 21607 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:58:40.966044 21607 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:58:40.966650 21607 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:58:40.970654 21607 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25d9f6acc04d4684a92ab68e4aeb4376: Generated new TSK 0
I20260927 03:58:40.970846 21607 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:58:41.095862 21596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:40.981504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36050 (local address 127.15.185.126:41385)
0927 03:58:40.981577 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:40.981579 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:40.981610 (+ 31us) server_negotiation.cc:407] Connection header received
0927 03:58:40.981696 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:40.981696 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:40.981710 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:40.981780 (+ 70us) server_negotiation.cc:226] Negotiated authn=JWT
0927 03:58:40.981903 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:40.982697 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:40.983612 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:40.983936 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:40.984010 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:40.984205 (+ 195us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 03:58:41.095624 (+111419us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 03:58:41.095801 (+ 177us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36050: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":48}
W20260927 03:58:41.095929 21596 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36050: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260927 03:58:41.096303 21622 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:58:40.981365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:41385 (local address 127.0.0.1:36050)
0927 03:58:40.981543 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:40.981558 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:40.981626 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:40.981786 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:40.981787 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:40.981788 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0927 03:58:40.981837 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:40.981840 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:40.982804 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:40.982805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:40.983528 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:40.983531 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:40.984039 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:40.984040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:40.984090 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:40.984096 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 03:58:41.095796 (+111700us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 03:58:41.095965 (+ 169us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.15.185.126:41385: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":100,"threads_started":1}
I20260927 03:58:41.096750 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 21526
2026-09-27T03:58:41Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (309 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (90 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (89 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:58:41 dist-test-slave-glgz krb5kdc[21634](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:58:41 dist-test-slave-glgz krb5kdc[21634](info): set up 2 sockets
Sep 27 03:58:41 dist-test-slave-glgz krb5kdc[21634](info): commencing operation
krb5kdc: starting...
W20260927 03:58:42.332051 16101 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Sep 27 03:58:42 dist-test-slave-glgz krb5kdc[21634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:58:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:58:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.185.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab.
Entry for principal kudu/127.15.185.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab.
I20260927 03:58:42.456115 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:45625
--webserver_interface=127.15.185.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:36659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.185.126:45625
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:58:42.536871 21647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:42.537081 21647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:42.537103 21647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:42.538184 21647 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:58:42.538218 21647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:42.538233 21647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:42.538245 21647 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:58:42.538257 21647 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:58:42.539692 21647 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:36659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.185.126:45625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:45625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:42.539887 21647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:42.540107 21647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:42.544917 21653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:42.544973 21652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:42.545092 21647 server_base.cc:1034] running on GCE node
W20260927 03:58:42.544917 21655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:42.545563 21647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:42.545847 21647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:42.547037 21647 hybrid_clock.cc:648] HybridClock initialized: now 1790481522547021 us; error 28 us; skew 500 ppm
Sep 27 03:58:42 dist-test-slave-glgz krb5kdc[21634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481522, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:42.549742 21647 init.cc:377] Logged in from keytab as kudu/127.15.185.126@KRBTEST.COM (short username kudu)
I20260927 03:58:42.550323 21647 webserver.cc:460] Webserver started at http://127.15.185.126:46481/ using document root <none> and password file <none>
I20260927 03:58:42.550575 21647 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:42.550643 21647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:42.550801 21647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:42.551923 21647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/minicluster-data/master-0/data/instance:
uuid: "2c0e764c8c7e4318b2904f5f9ad976e4"
format_stamp: "Formatted at 2026-09-27 03:58:42 on dist-test-slave-glgz"
server_key: "4510cc53732fcf9b0f32608a3a03c3e6"
server_key_iv: "f4d10472041799e10e2d43fb77f54f8b"
server_key_version: "encryptionkey@0"
I20260927 03:58:42.552286 21647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/minicluster-data/master-0/wal/instance:
uuid: "2c0e764c8c7e4318b2904f5f9ad976e4"
format_stamp: "Formatted at 2026-09-27 03:58:42 on dist-test-slave-glgz"
server_key: "4510cc53732fcf9b0f32608a3a03c3e6"
server_key_iv: "f4d10472041799e10e2d43fb77f54f8b"
server_key_version: "encryptionkey@0"
I20260927 03:58:42.553642 21647 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 03:58:42.554450 21662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:42.554625 21647 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:58:42.554721 21647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/minicluster-data/master-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/minicluster-data/master-0/wal
uuid: "2c0e764c8c7e4318b2904f5f9ad976e4"
format_stamp: "Formatted at 2026-09-27 03:58:42 on dist-test-slave-glgz"
server_key: "4510cc53732fcf9b0f32608a3a03c3e6"
server_key_iv: "f4d10472041799e10e2d43fb77f54f8b"
server_key_version: "encryptionkey@0"
I20260927 03:58:42.554803 21647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:42.563679 21647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:42.565732 21647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:42.565915 21647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:42.569885 21647 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.126:45625
I20260927 03:58:42.570004 21714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.126:45625 every 8 connection(s)
I20260927 03:58:42.570313 21647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
I20260927 03:58:42.570657 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 21647
I20260927 03:58:42.570760 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/minicluster-data/master-0/wal/instance
I20260927 03:58:42.570955 16101 external_mini_cluster.cc:1428] Setting key 6f3ae6795905e5b125184aa01029e9cc
I20260927 03:58:42.571673 21715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:42.574801 21715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c0e764c8c7e4318b2904f5f9ad976e4: Bootstrap starting.
I20260927 03:58:42.575507 21715 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c0e764c8c7e4318b2904f5f9ad976e4: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:42.575834 21715 log.cc:824] T 00000000000000000000000000000000 P 2c0e764c8c7e4318b2904f5f9ad976e4: Log is configured to *not* fsync() on all Append() calls
I20260927 03:58:42.576574 21715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c0e764c8c7e4318b2904f5f9ad976e4: No bootstrap required, opened a new log
Sep 27 03:58:42 dist-test-slave-glgz krb5kdc[21634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 03:58:42.578491 21715 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c0e764c8c7e4318b2904f5f9ad976e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c0e764c8c7e4318b2904f5f9ad976e4" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45625 } }
I20260927 03:58:42.578652 21715 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c0e764c8c7e4318b2904f5f9ad976e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:42.578712 21715 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c0e764c8c7e4318b2904f5f9ad976e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c0e764c8c7e4318b2904f5f9ad976e4, State: Initialized, Role: FOLLOWER
I20260927 03:58:42.578886 21715 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c0e764c8c7e4318b2904f5f9ad976e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c0e764c8c7e4318b2904f5f9ad976e4" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45625 } }
I20260927 03:58:42.578991 21715 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c0e764c8c7e4318b2904f5f9ad976e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:58:42.579028 21715 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c0e764c8c7e4318b2904f5f9ad976e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:58:42.579103 21715 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c0e764c8c7e4318b2904f5f9ad976e4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:42.579747 21715 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c0e764c8c7e4318b2904f5f9ad976e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c0e764c8c7e4318b2904f5f9ad976e4" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45625 } }
I20260927 03:58:42.579861 21715 leader_election.cc:302] T 00000000000000000000000000000000 P 2c0e764c8c7e4318b2904f5f9ad976e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c0e764c8c7e4318b2904f5f9ad976e4; no voters:
I20260927 03:58:42.580072 21715 leader_election.cc:288] T 00000000000000000000000000000000 P 2c0e764c8c7e4318b2904f5f9ad976e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:58:42.580103 21720 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c0e764c8c7e4318b2904f5f9ad976e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:58:42.580214 21720 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c0e764c8c7e4318b2904f5f9ad976e4 [term 1 LEADER]: Becoming Leader. State: Replica: 2c0e764c8c7e4318b2904f5f9ad976e4, State: Running, Role: LEADER
I20260927 03:58:42.580296 21720 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c0e764c8c7e4318b2904f5f9ad976e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c0e764c8c7e4318b2904f5f9ad976e4" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45625 } }
I20260927 03:58:42.580351 21715 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c0e764c8c7e4318b2904f5f9ad976e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:58:42.580705 21721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c0e764c8c7e4318b2904f5f9ad976e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c0e764c8c7e4318b2904f5f9ad976e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c0e764c8c7e4318b2904f5f9ad976e4" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45625 } } }
I20260927 03:58:42.580790 21721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c0e764c8c7e4318b2904f5f9ad976e4 [sys.catalog]: This master's current role is: LEADER
I20260927 03:58:42.581004 21722 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c0e764c8c7e4318b2904f5f9ad976e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c0e764c8c7e4318b2904f5f9ad976e4. Latest consensus state: current_term: 1 leader_uuid: "2c0e764c8c7e4318b2904f5f9ad976e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c0e764c8c7e4318b2904f5f9ad976e4" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45625 } } }
I20260927 03:58:42.581139 21722 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c0e764c8c7e4318b2904f5f9ad976e4 [sys.catalog]: This master's current role is: LEADER
I20260927 03:58:42.581305 21726 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:58:42.581848 21726 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:58:42.582540 21718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:42.571523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35840 (local address 127.15.185.126:45625)
0927 03:58:42.571696 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:42.571706 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:42.571740 (+ 34us) server_negotiation.cc:407] Connection header received
0927 03:58:42.572318 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:42.572327 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:42.572391 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:42.572545 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:42.573217 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:42.574617 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:42.575571 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:42.576173 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:42.576305 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:42.578660 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:42.578665 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:42.578665 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:42.578679 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:42.580870 (+ 2191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:42.581351 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:42.581354 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:42.581355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:42.581456 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:42.581708 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:42.581709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:42.581710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:42.581969 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:42.582143 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:42.582218 (+ 75us) server_negotiation.cc:299] Negotiation successful
0927 03:58:42.582356 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":82,"threads_started":1}
I20260927 03:58:42.584105 21726 catalog_manager.cc:1347] Generated new cluster ID: c4338b1976ff40959158422d0aeed201
I20260927 03:58:42.584160 21726 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:58:42.599658 21726 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:58:42.600375 21726 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:58:42.605537 21726 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c0e764c8c7e4318b2904f5f9ad976e4: Generated new TSK 0
I20260927 03:58:42.605726 21726 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.185.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab.
Entry for principal kudu/127.15.185.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab.
I20260927 03:58:42.634178 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.65:0
--local_ip_for_outbound_sockets=127.15.185.65
--webserver_interface=127.15.185.65
--webserver_port=0
--tserver_master_addrs=127.15.185.126:45625
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab
--principal=kudu/127.15.185.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:36659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:58:42.716163 21741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:42.716403 21741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:42.716450 21741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:42.717623 21741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:42.717689 21741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:42.717734 21741 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.65
I20260927 03:58:42.719264 21741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:36659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab
--principal=kudu/127.15.185.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.185.65
--webserver_port=0
--tserver_master_addrs=127.15.185.126:45625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.65
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:42.719538 21741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:42.719764 21741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:42.724677 21746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:42.724675 21747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:42.724838 21749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:42.725014 21741 server_base.cc:1034] running on GCE node
I20260927 03:58:42.725267 21741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:42.725505 21741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:42.726642 21741 hybrid_clock.cc:648] HybridClock initialized: now 1790481522726630 us; error 24 us; skew 500 ppm
Sep 27 03:58:42 dist-test-slave-glgz krb5kdc[21634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481522, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:42.729074 21741 init.cc:377] Logged in from keytab as kudu/127.15.185.65@KRBTEST.COM (short username kudu)
I20260927 03:58:42.729506 21741 webserver.cc:460] Webserver started at http://127.15.185.65:39385/ using document root <none> and password file <none>
I20260927 03:58:42.729681 21741 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:42.729720 21741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:42.729804 21741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:42.730793 21741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/minicluster-data/ts-0/data/instance:
uuid: "f1dc93c735e94ebea3aba6a939d0c433"
format_stamp: "Formatted at 2026-09-27 03:58:42 on dist-test-slave-glgz"
server_key: "ba5c51bd2983889a0d2c65efd5be4c6a"
server_key_iv: "077c21ef3a530a0b220d7dd7692d649a"
server_key_version: "encryptionkey@0"
I20260927 03:58:42.731155 21741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/minicluster-data/ts-0/wal/instance:
uuid: "f1dc93c735e94ebea3aba6a939d0c433"
format_stamp: "Formatted at 2026-09-27 03:58:42 on dist-test-slave-glgz"
server_key: "ba5c51bd2983889a0d2c65efd5be4c6a"
server_key_iv: "077c21ef3a530a0b220d7dd7692d649a"
server_key_version: "encryptionkey@0"
I20260927 03:58:42.732401 21741 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:58:42.733119 21756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:42.733297 21741 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:58:42.733398 21741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/minicluster-data/ts-0/wal
uuid: "f1dc93c735e94ebea3aba6a939d0c433"
format_stamp: "Formatted at 2026-09-27 03:58:42 on dist-test-slave-glgz"
server_key: "ba5c51bd2983889a0d2c65efd5be4c6a"
server_key_iv: "077c21ef3a530a0b220d7dd7692d649a"
server_key_version: "encryptionkey@0"
I20260927 03:58:42.733582 21741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:42.757568 21741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:42.759615 21741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:42.759789 21741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:42.760048 21741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:58:42.760370 21741 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:58:42.760429 21741 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:42.760473 21741 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:58:42.760501 21741 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:42.768280 21741 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.65:41993
I20260927 03:58:42.768363 21869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.65:41993 every 8 connection(s)
I20260927 03:58:42.768740 21741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
I20260927 03:58:42.769585 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 21741
I20260927 03:58:42.769695 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/minicluster-data/ts-0/wal/instance
I20260927 03:58:42.769927 16101 external_mini_cluster.cc:1428] Setting key 90767b9703a9a2b027064fc5ff946640
Sep 27 03:58:42 dist-test-slave-glgz krb5kdc[21634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481522, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.65@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.185.66@KRBTEST.COM; defaulting to no policy
I20260927 03:58:42.779542 21718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:42.770553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:47697 (local address 127.15.185.126:45625)
0927 03:58:42.770611 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:42.770613 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:42.770624 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:58:42.771291 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:42.771292 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:42.771306 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:42.771383 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:42.772163 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:42.772681 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:42.773844 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:42.774048 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:42.774103 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:42.776558 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:42.776560 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:42.776561 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:42.776569 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:42.778063 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:42.778515 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:42.778516 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:42.778516 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:42.778578 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:42.778840 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:42.778849 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:42.778849 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:42.779003 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:42.779129 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:42.779401 (+ 272us) server_negotiation.cc:299] Negotiation successful
0927 03:58:42.779477 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 03:58:42.779663 21872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:42.769564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45625 (local address 127.15.185.65:47697)
0927 03:58:42.769825 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:42.769871 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:42.771152 (+ 1281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:42.771498 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:42.771502 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:42.771867 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:42.772067 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:42.772072 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:42.772809 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:42.772810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:42.773744 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:42.773747 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:42.774137 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:42.774138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:42.774289 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:42.774808 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:42.774818 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:42.776454 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:42.778186 (+ 1732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:42.778192 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:42.778194 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:42.778420 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:42.778646 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:42.778647 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:42.778647 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:42.778759 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:42.779118 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:42.779124 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:42.779255 (+ 131us) client_negotiation.cc:769] Sending connection context
0927 03:58:42.779352 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 03:58:42.779474 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":82,"threads_started":1}
I20260927 03:58:42.780227 21870 heartbeater.cc:347] Connected to a master server at 127.15.185.126:45625
I20260927 03:58:42.780354 21870 heartbeater.cc:464] Registering TS with master...
I20260927 03:58:42.780529 21870 heartbeater.cc:511] Master 127.15.185.126:45625 requested a full tablet report, sending...
I20260927 03:58:42.781131 21679 ts_manager.cc:194] Registered new tserver with Master: f1dc93c735e94ebea3aba6a939d0c433 (127.15.185.65:41993)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.66@KRBTEST.COM" created.
I20260927 03:58:42.782094 21679 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.185.65@KRBTEST.COM'} at 127.15.185.65:47697
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab.
Entry for principal kudu/127.15.185.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab.
I20260927 03:58:42.798468 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.66:0
--local_ip_for_outbound_sockets=127.15.185.66
--webserver_interface=127.15.185.66
--webserver_port=0
--tserver_master_addrs=127.15.185.126:45625
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab
--principal=kudu/127.15.185.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:36659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:58:42.879760 21875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:42.879969 21875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:42.880000 21875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:42.881125 21875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:42.881161 21875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:42.881238 21875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.66
I20260927 03:58:42.882761 21875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:36659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab
--principal=kudu/127.15.185.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.185.66
--webserver_port=0
--tserver_master_addrs=127.15.185.126:45625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.66
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:42.882956 21875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:42.883152 21875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:42.888013 21883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:42.888047 21880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:42.888033 21881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:42.888096 21875 server_base.cc:1034] running on GCE node
I20260927 03:58:42.888410 21875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:42.888669 21875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:42.889859 21875 hybrid_clock.cc:648] HybridClock initialized: now 1790481522889841 us; error 38 us; skew 500 ppm
Sep 27 03:58:42 dist-test-slave-glgz krb5kdc[21634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481522, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:42.892438 21875 init.cc:377] Logged in from keytab as kudu/127.15.185.66@KRBTEST.COM (short username kudu)
I20260927 03:58:42.892930 21875 webserver.cc:460] Webserver started at http://127.15.185.66:43631/ using document root <none> and password file <none>
I20260927 03:58:42.893178 21875 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:42.893249 21875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:42.893399 21875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:42.894565 21875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/minicluster-data/ts-1/data/instance:
uuid: "f808bee6f9a24de0b1aa668c05241101"
format_stamp: "Formatted at 2026-09-27 03:58:42 on dist-test-slave-glgz"
server_key: "2465cf46f004f9178799ce93bc61fb53"
server_key_iv: "952306a412f4e17bd328aef7a4e40d0d"
server_key_version: "encryptionkey@0"
I20260927 03:58:42.894932 21875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/minicluster-data/ts-1/wal/instance:
uuid: "f808bee6f9a24de0b1aa668c05241101"
format_stamp: "Formatted at 2026-09-27 03:58:42 on dist-test-slave-glgz"
server_key: "2465cf46f004f9178799ce93bc61fb53"
server_key_iv: "952306a412f4e17bd328aef7a4e40d0d"
server_key_version: "encryptionkey@0"
I20260927 03:58:42.896283 21875 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:58:42.897024 21890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:42.897192 21875 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:42.897295 21875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/minicluster-data/ts-1/wal
uuid: "f808bee6f9a24de0b1aa668c05241101"
format_stamp: "Formatted at 2026-09-27 03:58:42 on dist-test-slave-glgz"
server_key: "2465cf46f004f9178799ce93bc61fb53"
server_key_iv: "952306a412f4e17bd328aef7a4e40d0d"
server_key_version: "encryptionkey@0"
I20260927 03:58:42.897368 21875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:42.909552 21875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:42.912022 21875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:42.912182 21875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:42.912418 21875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:58:42.912734 21875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:58:42.912787 21875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:42.912828 21875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:58:42.912856 21875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:42.918672 21875 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.66:40289
I20260927 03:58:42.918720 22003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.66:40289 every 8 connection(s)
I20260927 03:58:42.919034 21875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
I20260927 03:58:42.923424 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 21875
I20260927 03:58:42.923537 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/minicluster-data/ts-1/wal/instance
I20260927 03:58:42.923714 16101 external_mini_cluster.cc:1428] Setting key 0e4fe56cda2ed33dadb3e4b9964bd179
Sep 27 03:58:42 dist-test-slave-glgz krb5kdc[21634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481522, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.66@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 03:58:42.929507 21718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:42.919839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:37919 (local address 127.15.185.126:45625)
0927 03:58:42.919973 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:42.919975 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:42.920139 (+ 164us) server_negotiation.cc:407] Connection header received
0927 03:58:42.921115 (+ 976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:42.921117 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:42.921131 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:42.921179 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:42.921811 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:42.922330 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:42.923452 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:42.923716 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:42.923800 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:42.926623 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:42.926625 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:42.926625 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:42.926632 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:42.927992 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:42.928530 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:42.928531 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:42.928532 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:42.928583 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:42.928864 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:42.928864 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:42.928865 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:42.928966 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:42.929014 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:42.929371 (+ 357us) server_negotiation.cc:299] Negotiation successful
0927 03:58:42.929435 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260927 03:58:42.929844 22006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:42.919767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45625 (local address 127.15.185.66:37919)
0927 03:58:42.920001 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:42.920047 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:42.920991 (+ 944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:42.921250 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:42.921254 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:42.921520 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:42.921709 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:42.921713 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:42.922440 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:42.922441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:42.923363 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:42.923365 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:42.923823 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:42.923825 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:42.923985 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:42.924609 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:42.924620 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:42.926513 (+ 1893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:42.928105 (+ 1592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:42.928111 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:42.928114 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:42.928451 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:42.928646 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:42.928647 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:42.928647 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:42.928795 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:42.929032 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:42.929038 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:42.929213 (+ 175us) client_negotiation.cc:769] Sending connection context
0927 03:58:42.929337 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 03:58:42.929562 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":80,"threads_started":1}
I20260927 03:58:42.930320 22004 heartbeater.cc:347] Connected to a master server at 127.15.185.126:45625
I20260927 03:58:42.930430 22004 heartbeater.cc:464] Registering TS with master...
I20260927 03:58:42.930631 22004 heartbeater.cc:511] Master 127.15.185.126:45625 requested a full tablet report, sending...
I20260927 03:58:42.930985 21679 ts_manager.cc:194] Registered new tserver with Master: f808bee6f9a24de0b1aa668c05241101 (127.15.185.66:40289)
I20260927 03:58:42.931380 21679 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.185.66@KRBTEST.COM'} at 127.15.185.66:37919
WARNING: no policy specified for kudu/127.15.185.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab.
Entry for principal kudu/127.15.185.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab.
I20260927 03:58:42.951098 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.67:0
--local_ip_for_outbound_sockets=127.15.185.67
--webserver_interface=127.15.185.67
--webserver_port=0
--tserver_master_addrs=127.15.185.126:45625
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab
--principal=kudu/127.15.185.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:36659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:58:43.033046 22009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:43.033254 22009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:43.033278 22009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:43.034497 22009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:43.034539 22009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:43.034566 22009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.67
I20260927 03:58:43.036255 22009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:36659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab
--principal=kudu/127.15.185.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.185.67
--webserver_port=0
--tserver_master_addrs=127.15.185.126:45625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.67
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:43.036515 22009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:43.036767 22009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:43.041947 22014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:43.041963 22017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:43.041981 22015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:43.041982 22009 server_base.cc:1034] running on GCE node
I20260927 03:58:43.042268 22009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:43.042465 22009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:43.043604 22009 hybrid_clock.cc:648] HybridClock initialized: now 1790481523043591 us; error 28 us; skew 500 ppm
Sep 27 03:58:43 dist-test-slave-glgz krb5kdc[21634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481523, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:43.046056 22009 init.cc:377] Logged in from keytab as kudu/127.15.185.67@KRBTEST.COM (short username kudu)
I20260927 03:58:43.046505 22009 webserver.cc:460] Webserver started at http://127.15.185.67:41675/ using document root <none> and password file <none>
I20260927 03:58:43.046696 22009 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:43.046736 22009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:43.046824 22009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:43.047887 22009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/minicluster-data/ts-2/data/instance:
uuid: "3819e49d71ac47579a0bcb8bca246c3b"
format_stamp: "Formatted at 2026-09-27 03:58:43 on dist-test-slave-glgz"
server_key: "19c91ee42e031cd9d70b58b7dc9e2ee4"
server_key_iv: "a21e980d221fd100c6dc6581a40f80eb"
server_key_version: "encryptionkey@0"
I20260927 03:58:43.048179 22009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/minicluster-data/ts-2/wal/instance:
uuid: "3819e49d71ac47579a0bcb8bca246c3b"
format_stamp: "Formatted at 2026-09-27 03:58:43 on dist-test-slave-glgz"
server_key: "19c91ee42e031cd9d70b58b7dc9e2ee4"
server_key_iv: "a21e980d221fd100c6dc6581a40f80eb"
server_key_version: "encryptionkey@0"
I20260927 03:58:43.049405 22009 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 03:58:43.050105 22024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:43.050275 22009 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:58:43.050374 22009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/minicluster-data/ts-2/wal
uuid: "3819e49d71ac47579a0bcb8bca246c3b"
format_stamp: "Formatted at 2026-09-27 03:58:43 on dist-test-slave-glgz"
server_key: "19c91ee42e031cd9d70b58b7dc9e2ee4"
server_key_iv: "a21e980d221fd100c6dc6581a40f80eb"
server_key_version: "encryptionkey@0"
I20260927 03:58:43.050472 22009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:43.059633 22009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:43.061851 22009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:43.062008 22009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:43.062220 22009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:58:43.062517 22009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:58:43.062569 22009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:43.062613 22009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:58:43.062654 22009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:43.068846 22009 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.67:39863
I20260927 03:58:43.068883 22137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.67:39863 every 8 connection(s)
I20260927 03:58:43.069204 22009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
Sep 27 03:58:43 dist-test-slave-glgz krb5kdc[21634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481523, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.67@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 03:58:43.075686 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 22009
I20260927 03:58:43.075803 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/minicluster-data/ts-2/wal/instance
I20260927 03:58:43.076022 16101 external_mini_cluster.cc:1428] Setting key 33e334ce042936f3fd21729df6b404ce
I20260927 03:58:43.078831 21718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:43.069947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:56167 (local address 127.15.185.126:45625)
0927 03:58:43.070032 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:43.070034 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:43.070235 (+ 201us) server_negotiation.cc:407] Connection header received
0927 03:58:43.071189 (+ 954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:43.071190 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:43.071199 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:43.071261 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:43.071833 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:43.072316 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:43.073317 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:43.073495 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:43.073546 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:43.075772 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:43.075774 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:43.075774 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:43.075781 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:43.077448 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:43.077861 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:43.077862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:43.077863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:43.077931 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:43.078190 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:43.078191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:43.078191 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:43.078307 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:43.078384 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:43.078664 (+ 280us) server_negotiation.cc:299] Negotiation successful
0927 03:58:43.078740 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 03:58:43.079030 22140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:43.069955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45625 (local address 127.15.185.67:56167)
0927 03:58:43.070110 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:43.070151 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:43.071092 (+ 941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:43.071317 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:43.071320 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:43.071569 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:43.071757 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:43.071760 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:43.072397 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:43.072398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:43.073241 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:43.073243 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:43.073564 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:43.073565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:43.073658 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:43.074244 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:43.074253 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:43.075659 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:43.077544 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:43.077550 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:43.077552 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:43.077775 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:43.077998 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:43.077999 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:43.077999 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:43.078115 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:43.078373 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:43.078379 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:43.078526 (+ 147us) client_negotiation.cc:769] Sending connection context
0927 03:58:43.078631 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 03:58:43.078792 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":83,"threads_started":1}
I20260927 03:58:43.079453 22138 heartbeater.cc:347] Connected to a master server at 127.15.185.126:45625
I20260927 03:58:43.079560 22138 heartbeater.cc:464] Registering TS with master...
I20260927 03:58:43.079694 22138 heartbeater.cc:511] Master 127.15.185.126:45625 requested a full tablet report, sending...
I20260927 03:58:43.080075 21679 ts_manager.cc:194] Registered new tserver with Master: 3819e49d71ac47579a0bcb8bca246c3b (127.15.185.67:39863)
I20260927 03:58:43.080506 21679 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.185.67@KRBTEST.COM'} at 127.15.185.67:56167
I20260927 03:58:43.087765 16101 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/krb5kdc/test-admin.keytab.
Sep 27 03:58:43 dist-test-slave-glgz krb5kdc[21634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:43.104075 16101 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260927 03:58:43.106014 22145 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:58:43.105041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43280 (local address 127.15.185.65:41993)
0927 03:58:43.105222 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:43.105227 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:43.105238 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:58:43.105281 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:43.105284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:43.105300 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:43.105370 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:43.105907 (+ 537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43280: BlockingRecv error: recv got EOF from 127.0.0.1:43280 (error 108)
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":102,"threads_started":1}
W20260927 03:58:43.106138 16101 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/krb5kdc/krb5cc' of type FILE
Sep 27 03:58:43 dist-test-slave-glgz krb5kdc[21634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:43.107337 16101 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 03:58:43 dist-test-slave-glgz krb5kdc[21634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:43.108317 16101 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 03:58:43 dist-test-slave-glgz krb5kdc[21634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.185.65@KRBTEST.COM
I20260927 03:58:43.116602 22145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:43.108900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43284 (local address 127.15.185.65:41993)
0927 03:58:43.108962 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:43.108964 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:43.109043 (+ 79us) server_negotiation.cc:407] Connection header received
0927 03:58:43.109099 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:43.109099 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:43.109109 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:43.109157 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:43.109610 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:43.110334 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:43.111218 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:43.111556 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:43.111614 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:43.113776 (+ 2162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:43.113784 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:43.113784 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:43.113791 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:43.115614 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:43.115937 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:43.115940 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:43.115940 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:43.115994 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:43.116158 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:43.116159 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:43.116159 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:43.116289 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:43.116409 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:43.116464 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 03:58:43.116526 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260927 03:58:43.118606 22145 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:58:43.118052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43286 (local address 127.15.185.65:41993)
0927 03:58:43.118086 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:43.118088 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:43.118096 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:58:43.118153 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:43.118154 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:43.118165 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:43.118236 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:43.118531 (+ 295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43286: BlockingRecv error: recv got EOF from 127.0.0.1:43286 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260927 03:58:43.118677 16101 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/krb5kdc/krb5cc' of type FILE
Sep 27 03:58:43 dist-test-slave-glgz krb5kdc[21634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:43.119818 16101 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 03:58:43 dist-test-slave-glgz krb5kdc[21634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:43.120707 16101 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 03:58:43 dist-test-slave-glgz krb5kdc[21634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.185.65@KRBTEST.COM
I20260927 03:58:43.128688 22145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:43.121258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43294 (local address 127.15.185.65:41993)
0927 03:58:43.121320 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:43.121322 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:43.121332 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:58:43.121403 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:43.121404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:43.121415 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:43.121495 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:43.121887 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:43.122441 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:43.123264 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:43.123489 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:43.123602 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:43.125853 (+ 2251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:43.125855 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:43.125855 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:43.125863 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:43.127458 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:43.127847 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:43.127848 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:43.127849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:43.127929 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:43.128148 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:43.128149 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:43.128150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:43.128260 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:43.128379 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:43.128490 (+ 111us) server_negotiation.cc:299] Negotiation successful
0927 03:58:43.128574 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260927 03:58:43.130421 22145 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:58:43.129828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43304 (local address 127.15.185.65:41993)
0927 03:58:43.129870 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:43.129872 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:43.129965 (+ 93us) server_negotiation.cc:407] Connection header received
0927 03:58:43.130044 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:43.130045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:43.130054 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:43.130098 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:43.130377 (+ 279us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43304: BlockingRecv error: recv got EOF from 127.0.0.1:43304 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260927 03:58:43.130569 16101 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481498018407-16101-0/krb5kdc/krb5cc' of type FILE
Sep 27 03:58:43 dist-test-slave-glgz krb5kdc[21634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:43.131572 16101 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 03:58:43 dist-test-slave-glgz krb5kdc[21634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:43.132366 16101 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 03:58:43 dist-test-slave-glgz krb5kdc[21634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.185.65@KRBTEST.COM
I20260927 03:58:43.139714 22145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:43.132935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43308 (local address 127.15.185.65:41993)
0927 03:58:43.132972 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:43.132974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:43.133151 (+ 177us) server_negotiation.cc:407] Connection header received
0927 03:58:43.133173 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:43.133174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:43.133182 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:43.133225 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:43.133562 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:43.134076 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:43.134734 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:43.134936 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:43.134999 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:43.137246 (+ 2247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:43.137248 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:43.137249 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:43.137257 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:43.138689 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:43.138989 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:43.138990 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:43.138990 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:43.139039 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:43.139256 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:43.139256 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:43.139256 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:43.139345 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:43.139427 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:43.139513 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 03:58:43.139564 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 03:58:43.140328 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 21741
I20260927 03:58:43.145819 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 21875
I20260927 03:58:43.149919 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 22009
I20260927 03:58:43.154235 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 21647
2026-09-27T03:58:43Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1878 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:58:43 dist-test-slave-glgz krb5kdc[22158](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:58:43 dist-test-slave-glgz krb5kdc[22158](info): set up 2 sockets
Sep 27 03:58:43 dist-test-slave-glgz krb5kdc[22158](info): commencing operation
krb5kdc: starting...
W20260927 03:58:44.187681 16101 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Sep 27 03:58:44 dist-test-slave-glgz krb5kdc[22158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481524, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:58:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:58:44Z Disabled control of system clock
WARNING: no policy specified for oryx/127.15.185.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.185.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.185.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/krb5kdc/oryx_127.15.185.126.keytab.
Entry for principal oryx/127.15.185.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/krb5kdc/oryx_127.15.185.126.keytab.
I20260927 03:58:44.309904 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:43909
--webserver_interface=127.15.185.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/krb5kdc/oryx_127.15.185.126.keytab
--principal=oryx/127.15.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:45795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.185.126:43909
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:58:44.391952 22171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:44.392189 22171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:44.392213 22171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:44.393400 22171 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:58:44.393440 22171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:44.393455 22171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:44.393467 22171 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:58:44.393482 22171 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:58:44.395154 22171 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:45795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.185.126:43909
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/krb5kdc/oryx_127.15.185.126.keytab
--principal=oryx/127.15.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:43909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:44.395396 22171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:44.395643 22171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:44.400475 22177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:44.400502 22171 server_base.cc:1034] running on GCE node
W20260927 03:58:44.400475 22176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:44.400513 22179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:44.400899 22171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:44.401201 22171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:44.402352 22171 hybrid_clock.cc:648] HybridClock initialized: now 1790481524402341 us; error 33 us; skew 500 ppm
Sep 27 03:58:44 dist-test-slave-glgz krb5kdc[22158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481524, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:44.410235 22171 init.cc:377] Logged in from keytab as oryx/127.15.185.126@KRBTEST.COM (short username oryx)
I20260927 03:58:44.410727 22171 webserver.cc:460] Webserver started at http://127.15.185.126:37717/ using document root <none> and password file <none>
I20260927 03:58:44.410970 22171 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:44.411039 22171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:44.411234 22171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:44.412322 22171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/master-0/data/instance:
uuid: "1868152744144c8f97d2a16fc4d34438"
format_stamp: "Formatted at 2026-09-27 03:58:44 on dist-test-slave-glgz"
server_key: "a92eb38385f1c3cfde77aca2a8ff0c84"
server_key_iv: "ac255a045e2098335e7403ff1ccdeaec"
server_key_version: "encryptionkey@0"
I20260927 03:58:44.412699 22171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/master-0/wal/instance:
uuid: "1868152744144c8f97d2a16fc4d34438"
format_stamp: "Formatted at 2026-09-27 03:58:44 on dist-test-slave-glgz"
server_key: "a92eb38385f1c3cfde77aca2a8ff0c84"
server_key_iv: "ac255a045e2098335e7403ff1ccdeaec"
server_key_version: "encryptionkey@0"
I20260927 03:58:44.414135 22171 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:58:44.414913 22186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:44.415132 22171 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:58:44.415231 22171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/master-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/master-0/wal
uuid: "1868152744144c8f97d2a16fc4d34438"
format_stamp: "Formatted at 2026-09-27 03:58:44 on dist-test-slave-glgz"
server_key: "a92eb38385f1c3cfde77aca2a8ff0c84"
server_key_iv: "ac255a045e2098335e7403ff1ccdeaec"
server_key_version: "encryptionkey@0"
I20260927 03:58:44.415316 22171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:44.423981 22171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:44.426270 22171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:44.426460 22171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:44.430631 22171 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.126:43909
I20260927 03:58:44.430670 22238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.126:43909 every 8 connection(s)
I20260927 03:58:44.431015 22171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
I20260927 03:58:44.432089 22239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:44.434425 22239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1868152744144c8f97d2a16fc4d34438: Bootstrap starting.
I20260927 03:58:44.434770 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 22171
I20260927 03:58:44.434898 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/master-0/wal/instance
I20260927 03:58:44.435138 16101 external_mini_cluster.cc:1428] Setting key 830499a9afdbe9e5f45d868882d526ae
I20260927 03:58:44.435144 22239 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1868152744144c8f97d2a16fc4d34438: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:44.435433 22239 log.cc:824] T 00000000000000000000000000000000 P 1868152744144c8f97d2a16fc4d34438: Log is configured to *not* fsync() on all Append() calls
I20260927 03:58:44.436239 22239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1868152744144c8f97d2a16fc4d34438: No bootstrap required, opened a new log
I20260927 03:58:44.438484 22239 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1868152744144c8f97d2a16fc4d34438 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1868152744144c8f97d2a16fc4d34438" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 43909 } }
I20260927 03:58:44.438648 22239 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1868152744144c8f97d2a16fc4d34438 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:44.438683 22239 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1868152744144c8f97d2a16fc4d34438 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1868152744144c8f97d2a16fc4d34438, State: Initialized, Role: FOLLOWER
I20260927 03:58:44.438889 22239 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1868152744144c8f97d2a16fc4d34438 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1868152744144c8f97d2a16fc4d34438" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 43909 } }
I20260927 03:58:44.438995 22239 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1868152744144c8f97d2a16fc4d34438 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:58:44.439056 22239 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1868152744144c8f97d2a16fc4d34438 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:58:44.439109 22239 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1868152744144c8f97d2a16fc4d34438 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:44.439685 22239 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1868152744144c8f97d2a16fc4d34438 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1868152744144c8f97d2a16fc4d34438" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 43909 } }
I20260927 03:58:44.439787 22239 leader_election.cc:302] T 00000000000000000000000000000000 P 1868152744144c8f97d2a16fc4d34438 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1868152744144c8f97d2a16fc4d34438; no voters:
I20260927 03:58:44.439946 22239 leader_election.cc:288] T 00000000000000000000000000000000 P 1868152744144c8f97d2a16fc4d34438 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:58:44.439985 22244 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1868152744144c8f97d2a16fc4d34438 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:58:44.440178 22244 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1868152744144c8f97d2a16fc4d34438 [term 1 LEADER]: Becoming Leader. State: Replica: 1868152744144c8f97d2a16fc4d34438, State: Running, Role: LEADER
I20260927 03:58:44.440194 22239 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1868152744144c8f97d2a16fc4d34438 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:58:44.440313 22244 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1868152744144c8f97d2a16fc4d34438 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1868152744144c8f97d2a16fc4d34438" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 43909 } }
I20260927 03:58:44.440631 22244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1868152744144c8f97d2a16fc4d34438 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1868152744144c8f97d2a16fc4d34438. Latest consensus state: current_term: 1 leader_uuid: "1868152744144c8f97d2a16fc4d34438" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1868152744144c8f97d2a16fc4d34438" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 43909 } } }
I20260927 03:58:44.440752 22244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1868152744144c8f97d2a16fc4d34438 [sys.catalog]: This master's current role is: LEADER
I20260927 03:58:44.440641 22245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1868152744144c8f97d2a16fc4d34438 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1868152744144c8f97d2a16fc4d34438" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1868152744144c8f97d2a16fc4d34438" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 43909 } } }
I20260927 03:58:44.440991 22245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1868152744144c8f97d2a16fc4d34438 [sys.catalog]: This master's current role is: LEADER
I20260927 03:58:44.441908 22251 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:58:44.442441 22251 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 03:58:44 dist-test-slave-glgz krb5kdc[22158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481524, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM
I20260927 03:58:44.444017 22251 catalog_manager.cc:1347] Generated new cluster ID: 019b672ca57e471d9afa3c19afb55a10
I20260927 03:58:44.444100 22251 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:58:44.446762 22243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:44.435619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38300 (local address 127.15.185.126:43909)
0927 03:58:44.435796 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:44.435806 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:44.435842 (+ 36us) server_negotiation.cc:407] Connection header received
0927 03:58:44.436454 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:44.436462 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:44.436525 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:44.436679 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:44.437270 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:44.438671 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:44.439566 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:44.440184 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:44.440341 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:44.443080 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:44.443084 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:44.443084 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:44.443097 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:44.445155 (+ 2058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:44.445546 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:44.445549 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:44.445549 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:44.445656 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:44.445859 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:44.445860 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:44.445861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:44.446161 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:44.446313 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:44.446406 (+ 93us) server_negotiation.cc:299] Negotiation successful
0927 03:58:44.446542 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":107,"threads_started":1}
I20260927 03:58:44.452704 22251 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:58:44.453208 22251 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:58:44.463932 22251 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1868152744144c8f97d2a16fc4d34438: Generated new TSK 0
I20260927 03:58:44.464110 22251 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.15.185.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.185.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.185.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/krb5kdc/oryx_127.15.185.65.keytab.
Entry for principal oryx/127.15.185.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/krb5kdc/oryx_127.15.185.65.keytab.
I20260927 03:58:44.498376 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.65:0
--local_ip_for_outbound_sockets=127.15.185.65
--webserver_interface=127.15.185.65
--webserver_port=0
--tserver_master_addrs=127.15.185.126:43909
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/krb5kdc/oryx_127.15.185.65.keytab
--principal=oryx/127.15.185.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:45795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:58:44.599107 22265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:44.599350 22265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:44.599390 22265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:44.600545 22265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:44.600620 22265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:44.600667 22265 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.65
I20260927 03:58:44.602119 22265 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:45795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/krb5kdc/oryx_127.15.185.65.keytab
--principal=oryx/127.15.185.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.185.65
--webserver_port=0
--tserver_master_addrs=127.15.185.126:43909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22265
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.65
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:44.602392 22265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:44.602628 22265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:44.607762 22270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:44.607756 22273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:44.607757 22271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:44.607785 22265 server_base.cc:1034] running on GCE node
I20260927 03:58:44.608161 22265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:44.608359 22265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:44.609493 22265 hybrid_clock.cc:648] HybridClock initialized: now 1790481524609480 us; error 29 us; skew 500 ppm
Sep 27 03:58:44 dist-test-slave-glgz krb5kdc[22158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481524, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.185.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:44.611941 22265 init.cc:377] Logged in from keytab as oryx/127.15.185.65@KRBTEST.COM (short username oryx)
I20260927 03:58:44.612385 22265 webserver.cc:460] Webserver started at http://127.15.185.65:39655/ using document root <none> and password file <none>
I20260927 03:58:44.612634 22265 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:44.612681 22265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:44.612862 22265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:44.613992 22265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-0/data/instance:
uuid: "ce9010ea1dc94a0f91d6c8b10895c2f1"
format_stamp: "Formatted at 2026-09-27 03:58:44 on dist-test-slave-glgz"
server_key: "b31969b28bec3e81e8b243f2dcbbe371"
server_key_iv: "99b02b3fff23c358223e042d9a768da7"
server_key_version: "encryptionkey@0"
I20260927 03:58:44.614377 22265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-0/wal/instance:
uuid: "ce9010ea1dc94a0f91d6c8b10895c2f1"
format_stamp: "Formatted at 2026-09-27 03:58:44 on dist-test-slave-glgz"
server_key: "b31969b28bec3e81e8b243f2dcbbe371"
server_key_iv: "99b02b3fff23c358223e042d9a768da7"
server_key_version: "encryptionkey@0"
I20260927 03:58:44.615775 22265 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260927 03:58:44.616501 22280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:44.616688 22265 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:58:44.616791 22265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-0/wal
uuid: "ce9010ea1dc94a0f91d6c8b10895c2f1"
format_stamp: "Formatted at 2026-09-27 03:58:44 on dist-test-slave-glgz"
server_key: "b31969b28bec3e81e8b243f2dcbbe371"
server_key_iv: "99b02b3fff23c358223e042d9a768da7"
server_key_version: "encryptionkey@0"
I20260927 03:58:44.616864 22265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:44.625368 22265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:44.627641 22265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:44.627807 22265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:44.628038 22265 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:58:44.628342 22265 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:58:44.628396 22265 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:44.628437 22265 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:58:44.628472 22265 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:44.634011 22265 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.65:35459
I20260927 03:58:44.634085 22393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.65:35459 every 8 connection(s)
I20260927 03:58:44.634442 22265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
Sep 27 03:58:44 dist-test-slave-glgz krb5kdc[22158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481524, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.185.65@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM
I20260927 03:58:44.643702 22243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:44.635296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:40011 (local address 127.15.185.126:43909)
0927 03:58:44.635362 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:44.635364 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:44.635581 (+ 217us) server_negotiation.cc:407] Connection header received
0927 03:58:44.636037 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:44.636038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:44.636048 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:44.636097 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:44.636978 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:44.637475 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:44.638392 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:44.638600 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:44.638651 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:44.640803 (+ 2152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:44.640805 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:44.640806 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:44.640813 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:44.642250 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:44.642676 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:44.642677 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:44.642677 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:44.642734 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:44.643008 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:44.643009 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:44.643010 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:44.643175 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:44.643252 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:44.643525 (+ 273us) server_negotiation.cc:299] Negotiation successful
0927 03:58:44.643608 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 03:58:44.643867 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 22265
I20260927 03:58:44.643925 22396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:44.635287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:43909 (local address 127.15.185.65:40011)
0927 03:58:44.635454 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:44.635490 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:44.635954 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:44.636157 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:44.636161 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:44.636715 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:44.636903 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:44.636907 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:44.637561 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:44.637562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:44.638320 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:44.638322 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:44.638670 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:44.638671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:44.638771 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:44.639273 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:44.639280 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:44.640704 (+ 1424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:44.642352 (+ 1648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:44.642358 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:44.642360 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:44.642588 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:44.642802 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:44.642803 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:44.642803 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:44.642915 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:44.643243 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:44.643249 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:44.643390 (+ 141us) client_negotiation.cc:769] Sending connection context
0927 03:58:44.643504 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 03:58:44.643681 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":83,"threads_started":1}
I20260927 03:58:44.643971 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-0/wal/instance
I20260927 03:58:44.644193 16101 external_mini_cluster.cc:1428] Setting key 99334398a1c614abc29869d8f691c95b
I20260927 03:58:44.644548 22394 heartbeater.cc:347] Connected to a master server at 127.15.185.126:43909
I20260927 03:58:44.644665 22394 heartbeater.cc:464] Registering TS with master...
I20260927 03:58:44.644851 22394 heartbeater.cc:511] Master 127.15.185.126:43909 requested a full tablet report, sending...
I20260927 03:58:44.645460 22203 ts_manager.cc:194] Registered new tserver with Master: ce9010ea1dc94a0f91d6c8b10895c2f1 (127.15.185.65:35459)
I20260927 03:58:44.646389 22203 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.185.65@KRBTEST.COM'} at 127.15.185.65:40011
WARNING: no policy specified for oryx/127.15.185.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.185.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.185.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/krb5kdc/oryx_127.15.185.66.keytab.
Entry for principal oryx/127.15.185.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/krb5kdc/oryx_127.15.185.66.keytab.
I20260927 03:58:44.671466 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.66:0
--local_ip_for_outbound_sockets=127.15.185.66
--webserver_interface=127.15.185.66
--webserver_port=0
--tserver_master_addrs=127.15.185.126:43909
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/krb5kdc/oryx_127.15.185.66.keytab
--principal=oryx/127.15.185.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:45795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:58:44.752202 22399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:44.752436 22399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:44.752472 22399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:44.753612 22399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:44.753680 22399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:44.753724 22399 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.66
I20260927 03:58:44.755196 22399 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:45795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/krb5kdc/oryx_127.15.185.66.keytab
--principal=oryx/127.15.185.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.185.66
--webserver_port=0
--tserver_master_addrs=127.15.185.126:43909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22399
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.66
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:44.755477 22399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:44.755728 22399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:44.760594 22407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:44.760603 22405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:44.760594 22404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:44.760754 22399 server_base.cc:1034] running on GCE node
I20260927 03:58:44.761078 22399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:44.761336 22399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:44.762513 22399 hybrid_clock.cc:648] HybridClock initialized: now 1790481524762503 us; error 26 us; skew 500 ppm
Sep 27 03:58:44 dist-test-slave-glgz krb5kdc[22158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481524, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.185.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:44.765210 22399 init.cc:377] Logged in from keytab as oryx/127.15.185.66@KRBTEST.COM (short username oryx)
I20260927 03:58:44.765836 22399 webserver.cc:460] Webserver started at http://127.15.185.66:45427/ using document root <none> and password file <none>
I20260927 03:58:44.766081 22399 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:44.766180 22399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:44.766320 22399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:44.767678 22399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-1/data/instance:
uuid: "55d1a890d9f4473fa298231905647c4a"
format_stamp: "Formatted at 2026-09-27 03:58:44 on dist-test-slave-glgz"
server_key: "d091847a5c1b7506346785e1bd34bd3e"
server_key_iv: "5c42f956c8dd4d23e367becf85254f8a"
server_key_version: "encryptionkey@0"
I20260927 03:58:44.768100 22399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-1/wal/instance:
uuid: "55d1a890d9f4473fa298231905647c4a"
format_stamp: "Formatted at 2026-09-27 03:58:44 on dist-test-slave-glgz"
server_key: "d091847a5c1b7506346785e1bd34bd3e"
server_key_iv: "5c42f956c8dd4d23e367becf85254f8a"
server_key_version: "encryptionkey@0"
I20260927 03:58:44.770272 22399 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 03:58:44.771306 22414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:44.771495 22399 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:58:44.771628 22399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-1/wal
uuid: "55d1a890d9f4473fa298231905647c4a"
format_stamp: "Formatted at 2026-09-27 03:58:44 on dist-test-slave-glgz"
server_key: "d091847a5c1b7506346785e1bd34bd3e"
server_key_iv: "5c42f956c8dd4d23e367becf85254f8a"
server_key_version: "encryptionkey@0"
I20260927 03:58:44.771703 22399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:44.798367 22399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:44.800282 22399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:44.800432 22399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:44.800680 22399 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:58:44.800993 22399 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:58:44.801028 22399 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:44.801052 22399 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:58:44.801100 22399 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:44.807032 22399 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.66:40143
I20260927 03:58:44.807112 22527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.66:40143 every 8 connection(s)
I20260927 03:58:44.807477 22399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
Sep 27 03:58:44 dist-test-slave-glgz krb5kdc[22158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481524, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.185.66@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM
I20260927 03:58:44.816402 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 22399
I20260927 03:58:44.816437 22243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:44.808245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:60161 (local address 127.15.185.126:43909)
0927 03:58:44.808292 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:44.808294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:44.808491 (+ 197us) server_negotiation.cc:407] Connection header received
0927 03:58:44.808968 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:44.808968 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:44.808977 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:44.809041 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:44.809982 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:44.810483 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:44.811581 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:44.811767 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:44.811825 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:44.813801 (+ 1976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:44.813803 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:44.813803 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:44.813810 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:44.815281 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:44.815623 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:44.815624 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:44.815624 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:44.815674 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:44.815867 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:44.815868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:44.815868 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:44.815966 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:44.816029 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:44.816251 (+ 222us) server_negotiation.cc:299] Negotiation successful
0927 03:58:44.816330 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 03:58:44.816527 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-1/wal/instance
I20260927 03:58:44.816514 22530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:44.808203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:43909 (local address 127.15.185.66:60161)
0927 03:58:44.808359 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:44.808403 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:44.808886 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:44.809096 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:44.809099 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:44.809669 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:44.809864 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:44.809868 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:44.810580 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:44.810581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:44.811496 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:44.811499 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:44.811840 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:44.811841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:44.811934 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:44.812379 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:44.812387 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:44.813717 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:44.815347 (+ 1630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:44.815352 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:44.815354 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:44.815560 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:44.815728 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:44.815728 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:44.815729 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:44.815817 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:44.816015 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:44.816019 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:44.816126 (+ 107us) client_negotiation.cc:769] Sending connection context
0927 03:58:44.816205 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 03:58:44.816318 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":89,"threads_started":1}
I20260927 03:58:44.816735 16101 external_mini_cluster.cc:1428] Setting key fabbae5076315f2c1e4dafcb971e9714
I20260927 03:58:44.817174 22528 heartbeater.cc:347] Connected to a master server at 127.15.185.126:43909
I20260927 03:58:44.817287 22528 heartbeater.cc:464] Registering TS with master...
I20260927 03:58:44.817479 22528 heartbeater.cc:511] Master 127.15.185.126:43909 requested a full tablet report, sending...
I20260927 03:58:44.817883 22203 ts_manager.cc:194] Registered new tserver with Master: 55d1a890d9f4473fa298231905647c4a (127.15.185.66:40143)
I20260927 03:58:44.818470 22203 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.185.66@KRBTEST.COM'} at 127.15.185.66:60161
WARNING: no policy specified for oryx/127.15.185.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.185.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.185.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/krb5kdc/oryx_127.15.185.67.keytab.
Entry for principal oryx/127.15.185.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/krb5kdc/oryx_127.15.185.67.keytab.
I20260927 03:58:44.843286 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.67:0
--local_ip_for_outbound_sockets=127.15.185.67
--webserver_interface=127.15.185.67
--webserver_port=0
--tserver_master_addrs=127.15.185.126:43909
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/krb5kdc/oryx_127.15.185.67.keytab
--principal=oryx/127.15.185.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:45795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:58:44.923615 22533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:44.923822 22533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:44.923846 22533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:44.925032 22533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:44.925074 22533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:44.925101 22533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.67
I20260927 03:58:44.926537 22533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:45795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/krb5kdc/oryx_127.15.185.67.keytab
--principal=oryx/127.15.185.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.185.67
--webserver_port=0
--tserver_master_addrs=127.15.185.126:43909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.67
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:44.926785 22533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:44.927030 22533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:44.932265 22539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:44.932265 22541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:44.932281 22533 server_base.cc:1034] running on GCE node
W20260927 03:58:44.932288 22538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:44.932698 22533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:44.932991 22533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:44.934181 22533 hybrid_clock.cc:648] HybridClock initialized: now 1790481524934165 us; error 39 us; skew 500 ppm
Sep 27 03:58:44 dist-test-slave-glgz krb5kdc[22158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481524, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.185.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:44.936884 22533 init.cc:377] Logged in from keytab as oryx/127.15.185.67@KRBTEST.COM (short username oryx)
I20260927 03:58:44.937357 22533 webserver.cc:460] Webserver started at http://127.15.185.67:45851/ using document root <none> and password file <none>
I20260927 03:58:44.937613 22533 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:44.937686 22533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:44.937845 22533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:44.938942 22533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-2/data/instance:
uuid: "20ea15da3bd74f2498765c4f9733de16"
format_stamp: "Formatted at 2026-09-27 03:58:44 on dist-test-slave-glgz"
server_key: "6334a29d4bd30fdedca82594f71b52f5"
server_key_iv: "4f77e68d5323fdb41a7efa9b2b687c5d"
server_key_version: "encryptionkey@0"
I20260927 03:58:44.939424 22533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-2/wal/instance:
uuid: "20ea15da3bd74f2498765c4f9733de16"
format_stamp: "Formatted at 2026-09-27 03:58:44 on dist-test-slave-glgz"
server_key: "6334a29d4bd30fdedca82594f71b52f5"
server_key_iv: "4f77e68d5323fdb41a7efa9b2b687c5d"
server_key_version: "encryptionkey@0"
I20260927 03:58:44.941088 22533 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:58:44.941922 22548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:44.942133 22533 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:58:44.942251 22533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-2/wal
uuid: "20ea15da3bd74f2498765c4f9733de16"
format_stamp: "Formatted at 2026-09-27 03:58:44 on dist-test-slave-glgz"
server_key: "6334a29d4bd30fdedca82594f71b52f5"
server_key_iv: "4f77e68d5323fdb41a7efa9b2b687c5d"
server_key_version: "encryptionkey@0"
I20260927 03:58:44.942338 22533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:44.961805 22533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:44.963903 22533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:44.964093 22533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:44.964304 22533 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:58:44.964643 22533 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:58:44.964700 22533 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:44.964745 22533 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:58:44.964769 22533 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:44.970811 22533 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.67:33539
I20260927 03:58:44.970852 22661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.67:33539 every 8 connection(s)
I20260927 03:58:44.971236 22533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
Sep 27 03:58:44 dist-test-slave-glgz krb5kdc[22158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481524, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.185.67@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM
I20260927 03:58:44.978191 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 22533
I20260927 03:58:44.978287 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-2/wal/instance
I20260927 03:58:44.978505 16101 external_mini_cluster.cc:1428] Setting key 491e88b761f925f4f6820fbedd3178df
I20260927 03:58:44.980942 22243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:44.971964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:44087 (local address 127.15.185.126:43909)
0927 03:58:44.972021 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:44.972023 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:44.972232 (+ 209us) server_negotiation.cc:407] Connection header received
0927 03:58:44.972752 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:44.972754 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:44.972766 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:44.972822 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:44.973794 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:44.974384 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:44.975521 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:44.975736 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:44.975822 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:44.977889 (+ 2067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:44.977891 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:44.977891 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:44.977897 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:44.979736 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:44.980117 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:44.980118 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:44.980118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:44.980172 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:44.980383 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:44.980385 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:44.980385 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:44.980502 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:44.980576 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:44.980813 (+ 237us) server_negotiation.cc:299] Negotiation successful
0927 03:58:44.980879 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 03:58:44.981171 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:44.971923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:43909 (local address 127.15.185.67:44087)
0927 03:58:44.972105 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:44.972151 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:44.972651 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:44.972894 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:44.972899 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:44.973488 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:44.973679 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:44.973683 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:44.974475 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:44.974476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:44.975439 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:44.975441 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:44.975830 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:44.975831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:44.975956 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:44.976410 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:44.976418 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:44.977804 (+ 1386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:44.979797 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:44.979802 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:44.979804 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:44.980049 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:44.980218 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:44.980219 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:44.980219 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:44.980308 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:44.980582 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:44.980586 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:44.980701 (+ 115us) client_negotiation.cc:769] Sending connection context
0927 03:58:44.980785 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 03:58:44.980905 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":89,"threads_started":1}
I20260927 03:58:44.981658 22662 heartbeater.cc:347] Connected to a master server at 127.15.185.126:43909
I20260927 03:58:44.981741 22662 heartbeater.cc:464] Registering TS with master...
I20260927 03:58:44.981880 22662 heartbeater.cc:511] Master 127.15.185.126:43909 requested a full tablet report, sending...
I20260927 03:58:44.982223 22203 ts_manager.cc:194] Registered new tserver with Master: 20ea15da3bd74f2498765c4f9733de16 (127.15.185.67:33539)
I20260927 03:58:44.982553 22203 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.185.67@KRBTEST.COM'} at 127.15.185.67:44087
I20260927 03:58:44.990473 16101 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.15.185.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.185.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.185.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/krb5kdc/oryx_127.15.185.125.keytab.
Entry for principal oryx/127.15.185.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/krb5kdc/oryx_127.15.185.125.keytab.
I20260927 03:58:45.018458 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.125:36857
--webserver_interface=127.15.185.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/krb5kdc/oryx_127.15.185.125.keytab
--principal=oryx/127.15.185.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:45795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.185.126:43909,127.15.185.125:36857
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:58:45.099444 22667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:45.099665 22667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:45.099689 22667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:45.100888 22667 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:58:45.100936 22667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:45.100950 22667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:45.101011 22667 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:58:45.101030 22667 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:58:45.102504 22667 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:45795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.15.185.126:43909,127.15.185.125:36857
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/krb5kdc/oryx_127.15.185.125.keytab
--principal=oryx/127.15.185.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.125:36857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.15.185.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22667
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
W20260927 03:58:45.102612 22667 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.15.185.126:43909,127.15.185.125:36857'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260927 03:58:45.102809 22667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:45.103086 22667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:45.107942 22675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:45.107957 22672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:45.107942 22673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:45.108340 22667 server_base.cc:1034] running on GCE node
I20260927 03:58:45.108582 22667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:45.108870 22667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:45.110015 22667 hybrid_clock.cc:648] HybridClock initialized: now 1790481525109994 us; error 40 us; skew 500 ppm
Sep 27 03:58:45 dist-test-slave-glgz krb5kdc[22158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481525, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.185.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:45.112681 22667 init.cc:377] Logged in from keytab as oryx/127.15.185.125@KRBTEST.COM (short username oryx)
I20260927 03:58:45.113209 22667 webserver.cc:460] Webserver started at http://127.15.185.125:45727/ using document root <none> and password file <none>
I20260927 03:58:45.113451 22667 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:45.113497 22667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:45.113667 22667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:45.114778 22667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/master-1/data/instance:
uuid: "87685032f24e44c59ad7c48aaff26803"
format_stamp: "Formatted at 2026-09-27 03:58:45 on dist-test-slave-glgz"
server_key: "b478e18f7041855f24486a118b358ec6"
server_key_iv: "4ee8c402259078a2a8ce8c6f196c0c00"
server_key_version: "encryptionkey@0"
I20260927 03:58:45.115175 22667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/master-1/wal/instance:
uuid: "87685032f24e44c59ad7c48aaff26803"
format_stamp: "Formatted at 2026-09-27 03:58:45 on dist-test-slave-glgz"
server_key: "b478e18f7041855f24486a118b358ec6"
server_key_iv: "4ee8c402259078a2a8ce8c6f196c0c00"
server_key_version: "encryptionkey@0"
I20260927 03:58:45.116489 22667 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:58:45.117177 22682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:45.117340 22667 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:58:45.117434 22667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/master-1/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/master-1/wal
uuid: "87685032f24e44c59ad7c48aaff26803"
format_stamp: "Formatted at 2026-09-27 03:58:45 on dist-test-slave-glgz"
server_key: "b478e18f7041855f24486a118b358ec6"
server_key_iv: "4ee8c402259078a2a8ce8c6f196c0c00"
server_key_version: "encryptionkey@0"
I20260927 03:58:45.117533 22667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:45.151284 22667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:45.153841 22667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:45.154026 22667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:45.158237 22667 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.125:36857
I20260927 03:58:45.158290 22734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.125:36857 every 8 connection(s)
I20260927 03:58:45.158676 22667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/master-1/data/info.pb
I20260927 03:58:45.159706 22735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:45.161221 22735 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.185.126" port: 43909 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:58:45.163458 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 22667
Sep 27 03:58:45 dist-test-slave-glgz krb5kdc[22158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481525, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.185.125@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM
Sep 27 03:58:45 dist-test-slave-glgz krb5kdc[22158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481524, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.185.125@KRBTEST.COM
I20260927 03:58:45.170513 22243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:45.161946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38308 (local address 127.15.185.126:43909)
0927 03:58:45.162013 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:45.162015 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:45.162239 (+ 224us) server_negotiation.cc:407] Connection header received
0927 03:58:45.162669 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:45.162670 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:45.162679 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:45.162739 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:45.163427 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.164128 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.165132 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.165332 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.165412 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:45.167493 (+ 2081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:45.167495 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:45.167495 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:45.167503 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:45.168938 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:45.169405 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:45.169406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:45.169406 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:45.169475 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:45.169740 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:45.169741 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:45.169741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:45.169942 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:45.170014 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:45.170281 (+ 267us) server_negotiation.cc:299] Negotiation successful
0927 03:58:45.170363 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 03:58:45.170665 22737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:45.161944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:43909 (local address 127.0.0.1:38308)
0927 03:58:45.162105 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:45.162147 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:45.162590 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:45.162790 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:45.162793 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:45.163135 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:45.163333 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:45.163337 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.164232 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.164233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:45.165025 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:45.165027 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.165420 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.165421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:45.165551 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:45.166061 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:45.166069 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:45.167399 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:45.169013 (+ 1614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:45.169019 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:45.169021 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:45.169323 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:45.169569 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:45.169570 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:45.169570 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:45.169671 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:45.170011 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:45.170016 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:45.170132 (+ 116us) client_negotiation.cc:769] Sending connection context
0927 03:58:45.170218 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 03:58:45.170375 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":92,"threads_started":1}
I20260927 03:58:45.171271 22735 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.185.125" port: 36857 } has no permanent_uuid. Determining permanent_uuid...
I20260927 03:58:45.172493 22739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:45.163939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50552 (local address 127.15.185.125:36857)
0927 03:58:45.164116 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:45.164120 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:45.164143 (+ 23us) server_negotiation.cc:407] Connection header received
0927 03:58:45.164167 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:45.164170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:45.164188 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:45.164256 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:45.164821 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.165526 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.166459 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.166762 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.166843 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:45.169532 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:45.169539 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:45.169539 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:45.169546 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:45.171290 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:45.171701 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:45.171704 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:45.171704 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:45.171764 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:45.171957 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:45.171958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:45.171959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:45.172099 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:45.172273 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:45.172315 (+ 42us) server_negotiation.cc:299] Negotiation successful
0927 03:58:45.172388 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":87,"threads_started":1}
Sep 27 03:58:45 dist-test-slave-glgz krb5kdc[22158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481525, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.185.125@KRBTEST.COM for oryx/127.15.185.125@KRBTEST.COM
I20260927 03:58:45.177655 22740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:45.171601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50558 (local address 127.15.185.125:36857)
0927 03:58:45.171729 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:45.171731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:45.171737 (+ 6us) server_negotiation.cc:407] Connection header received
0927 03:58:45.171752 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:45.171752 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:45.171761 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:45.171843 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:45.172075 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.172614 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.173380 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.173590 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.173645 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:45.175252 (+ 1607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:45.175254 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:45.175254 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:45.175260 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:45.176842 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:45.177114 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:45.177115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:45.177115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:45.177155 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:45.177286 (+ 131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:45.177287 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:45.177287 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:45.177365 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:45.177408 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:45.177518 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 03:58:45.177578 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":91,"threads_started":1}
I20260927 03:58:45.177781 22737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:45.171524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.125:36857 (local address 127.0.0.1:50558)
0927 03:58:45.171577 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:45.171588 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:45.171627 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:45.171844 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:45.171845 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:45.171978 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:45.172003 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:45.172004 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.172690 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.172691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:45.173316 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:45.173318 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.173652 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.173653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:45.173691 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:45.173996 (+ 305us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:45.174001 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:45.175145 (+ 1144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:45.176905 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:45.176906 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:45.176906 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:45.177060 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:45.177204 (+ 144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:45.177205 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:45.177205 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:45.177251 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:45.177412 (+ 161us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:45.177412 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:45.177443 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 03:58:45.177491 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 03:58:45.177639 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":43}
I20260927 03:58:45.179091 22735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803: Bootstrap starting.
I20260927 03:58:45.179705 22735 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:45.179986 22735 log.cc:824] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803: Log is configured to *not* fsync() on all Append() calls
I20260927 03:58:45.180673 22735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803: No bootstrap required, opened a new log
I20260927 03:58:45.182058 22735 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1868152744144c8f97d2a16fc4d34438" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 43909 } } peers { permanent_uuid: "87685032f24e44c59ad7c48aaff26803" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 36857 } }
I20260927 03:58:45.182178 22735 raft_consensus.cc:374] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:45.182199 22735 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87685032f24e44c59ad7c48aaff26803, State: Initialized, Role: FOLLOWER
I20260927 03:58:45.182350 22735 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1868152744144c8f97d2a16fc4d34438" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 43909 } } peers { permanent_uuid: "87685032f24e44c59ad7c48aaff26803" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 36857 } }
I20260927 03:58:45.182603 22742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1868152744144c8f97d2a16fc4d34438" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 43909 } } peers { permanent_uuid: "87685032f24e44c59ad7c48aaff26803" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 36857 } } }
I20260927 03:58:45.182638 22735 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:58:45.182699 22742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 03:58:45.184258 22753 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:58:45.184311 22753 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:58:45.185803 22667 master_runner.cc:428] Detected that this master 87685032f24e44c59ad7c48aaff26803 is joining an existing cluster
I20260927 03:58:45.185870 22667 master_runner.cc:432] Initiating AddMaster RPC to add 127.15.185.125:36857
I20260927 03:58:45.193928 22243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:45.186922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38316 (local address 127.15.185.126:43909)
0927 03:58:45.186972 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:45.186974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:45.187150 (+ 176us) server_negotiation.cc:407] Connection header received
0927 03:58:45.187258 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:45.187259 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:45.187270 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:45.187327 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:45.187746 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.188538 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.189611 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.189787 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.189867 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:45.190922 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:45.190924 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:45.190924 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:45.190932 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:45.192590 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:45.193005 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:45.193007 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:45.193007 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:45.193070 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:45.193326 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:45.193328 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:45.193328 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:45.193467 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:45.193509 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:45.193737 (+ 228us) server_negotiation.cc:299] Negotiation successful
0927 03:58:45.193812 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 03:58:45.194223 22759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:45.186880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:43909 (local address 127.0.0.1:38316)
0927 03:58:45.187081 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:45.187093 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:45.187176 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:45.187332 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:45.187333 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:45.187611 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:45.187668 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:45.187670 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.188629 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.188630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:45.189520 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:45.189522 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.189886 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.189887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:45.189954 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:45.190415 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:45.190420 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:45.190834 (+ 414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:45.192682 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:45.192683 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:45.192683 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:45.192929 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:45.193146 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:45.193147 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:45.193148 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:45.193219 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:45.193573 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:45.193579 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:45.193618 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 03:58:45.194022 (+ 404us) client_negotiation.cc:240] Negotiation successful
0927 03:58:45.194121 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":126,"threads_started":1}
I20260927 03:58:45.194944 22740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:45.187108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50574 (local address 127.15.185.125:36857)
0927 03:58:45.187144 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:45.187146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:45.187360 (+ 214us) server_negotiation.cc:407] Connection header received
0927 03:58:45.187378 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:45.187379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:45.187387 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:45.187438 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:45.187854 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.188550 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.190542 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.190841 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.190911 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:45.192155 (+ 1244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:45.192156 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:45.192156 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:45.192162 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:45.193629 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:45.193992 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:45.193993 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:45.193993 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:45.194050 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:45.194402 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:45.194403 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:45.194403 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:45.194532 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:45.194602 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:45.194753 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 03:58:45.194840 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 03:58:45.195086 22760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:45.186949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.125:36857 (local address 127.0.0.1:50574)
0927 03:58:45.187160 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:45.187170 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:45.187250 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:45.187458 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:45.187459 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:45.187709 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:45.187781 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:45.187783 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.189630 (+ 1847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.189631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:45.190458 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:45.190460 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.191119 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.191120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:45.191200 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:45.191624 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:45.191628 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:45.192056 (+ 428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:45.193728 (+ 1672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:45.193728 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:45.193729 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:45.193903 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:45.194246 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:45.194247 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:45.194247 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:45.194330 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:45.194593 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:45.194593 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:45.194634 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 03:58:45.194781 (+ 147us) client_negotiation.cc:240] Negotiation successful
0927 03:58:45.194901 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":24,"thread_start_us":88,"threads_started":1}
Sep 27 03:58:45 dist-test-slave-glgz krb5kdc[22158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481524, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.185.126@KRBTEST.COM for oryx/127.15.185.125@KRBTEST.COM
I20260927 03:58:45.204093 22740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:45.196928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50584 (local address 127.15.185.125:36857)
0927 03:58:45.196966 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:45.196968 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:45.197128 (+ 160us) server_negotiation.cc:407] Connection header received
0927 03:58:45.197186 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:45.197187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:45.197196 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:45.197258 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:45.197704 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.198416 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.199184 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.199475 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.199554 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:45.201599 (+ 2045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:45.201600 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:45.201601 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:45.201606 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:45.202940 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:45.203366 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:45.203366 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:45.203367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:45.203414 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:45.203632 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:45.203633 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:45.203633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:45.203725 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:45.203773 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:45.203973 (+ 200us) server_negotiation.cc:299] Negotiation successful
0927 03:58:45.204033 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 03:58:45.204166 22761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:45.196871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.125:36857 (local address 127.0.0.1:50584)
0927 03:58:45.197029 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:45.197060 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:45.197121 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:45.197261 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:45.197263 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:45.197586 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:45.197630 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:45.197632 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.198493 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.198494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:45.199104 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:45.199107 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.199552 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.199553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:45.199605 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:45.200061 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:45.200066 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:45.201498 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:45.203021 (+ 1523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:45.203027 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:45.203029 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:45.203296 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:45.203500 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:45.203501 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:45.203501 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:45.203575 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:45.203786 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:45.203793 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:45.203848 (+ 55us) client_negotiation.cc:769] Sending connection context
0927 03:58:45.203937 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 03:58:45.204052 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":70,"threads_started":1}
I20260927 03:58:45.205027 22203 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.15.185.125:36857: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "87685032f24e44c59ad7c48aaff26803"
member_type: NON_VOTER
last_known_addr {
host: "127.15.185.125"
port: 36857
}
attrs {
promote: true
}
}
dest_uuid: "1868152744144c8f97d2a16fc4d34438"
cas_config_opid_index: -1
I20260927 03:58:45.205343 22203 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1868152744144c8f97d2a16fc4d34438 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1868152744144c8f97d2a16fc4d34438" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 43909 } } peers { permanent_uuid: "87685032f24e44c59ad7c48aaff26803" member_type: NON_VOTER last_known_addr { host: "127.15.185.125" port: 36857 } attrs { promote: true } }
I20260927 03:58:45.205830 22763 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1868152744144c8f97d2a16fc4d34438 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 87685032f24e44c59ad7c48aaff26803 (127.15.185.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1868152744144c8f97d2a16fc4d34438" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 43909 } } peers { permanent_uuid: "87685032f24e44c59ad7c48aaff26803" member_type: NON_VOTER last_known_addr { host: "127.15.185.125" port: 36857 } attrs { promote: true } } }
I20260927 03:58:45.206130 22709 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:45.206988 22763 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.15.185.125:36857
I20260927 03:58:45.207034 22709 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803 [term 1 FOLLOWER]: Refusing update from remote peer 1868152744144c8f97d2a16fc4d34438: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260927 03:58:45.207170 22763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1868152744144c8f97d2a16fc4d34438 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1868152744144c8f97d2a16fc4d34438" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1868152744144c8f97d2a16fc4d34438" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 43909 } } peers { permanent_uuid: "87685032f24e44c59ad7c48aaff26803" member_type: NON_VOTER last_known_addr { host: "127.15.185.125" port: 36857 } attrs { promote: true } } }
I20260927 03:58:45.207248 22763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1868152744144c8f97d2a16fc4d34438 [sys.catalog]: This master's current role is: LEADER
I20260927 03:58:45.207289 22667 master.cc:537] Master@127.15.185.125:36857 shutting down...
I20260927 03:58:45.207551 22763 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1868152744144c8f97d2a16fc4d34438 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87685032f24e44c59ad7c48aaff26803" member_type: NON_VOTER last_known_addr { host: "127.15.185.125" port: 36857 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 03:58:45.208263 22187 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260927 03:58:45.208647 22187 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1868152744144c8f97d2a16fc4d34438 -> Peer 87685032f24e44c59ad7c48aaff26803 (127.15.185.125:36857): Couldn't send request to peer 87685032f24e44c59ad7c48aaff26803. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 03:58:45.210346 22667 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:58:45.210419 22667 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:58:45.210494 22667 tablet_replica.cc:330] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803: stopping tablet replica
I20260927 03:58:45.223223 22667 master.cc:552] Master@127.15.185.125:36857 shutdown complete.
I20260927 03:58:45.223284 22667 master_runner.cc:305] Clearing existing system tablet
I20260927 03:58:45.223443 22667 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:45.223490 22667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:45.224226 22667 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:58:45.224931 22767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:45.225098 22667 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:45.225160 22667 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:58:45.225185 22667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/master-1/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/master-1/wal
uuid: "87685032f24e44c59ad7c48aaff26803"
format_stamp: "Formatted at 2026-09-27 03:58:45 on dist-test-slave-glgz"
server_key: "b478e18f7041855f24486a118b358ec6"
server_key_iv: "4ee8c402259078a2a8ce8c6f196c0c00"
server_key_version: "encryptionkey@0"
I20260927 03:58:45.225965 22667 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:58:45.227582 22667 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 03:58:45.227654 22667 log.cc:1196] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803: Deleting WAL directory at /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260927 03:58:45.227861 22667 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803: Deleting consensus metadata
I20260927 03:58:45.228016 22667 master_runner.cc:315] Copying system tablet from 127.15.185.126:43909
I20260927 03:58:45.228617 22667 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803: tablet copy: Beginning tablet copy session from remote peer at address 127.15.185.126:43909
I20260927 03:58:45.235605 22773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:45.228906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:43909 (local address 127.0.0.1:38318)
0927 03:58:45.229060 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:45.229079 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:45.229134 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:45.229273 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:45.229274 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:45.229417 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:45.229473 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:45.229474 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.230390 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.230391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:45.230919 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:45.230920 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.231440 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.231441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:45.231492 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:45.231977 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:45.231981 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:45.232266 (+ 285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:45.234062 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:45.234063 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:45.234063 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:45.234249 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:45.234607 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:45.234608 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:45.234609 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:45.234677 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:45.235263 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:45.235264 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:45.235322 (+ 58us) client_negotiation.cc:769] Sending connection context
0927 03:58:45.235407 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 03:58:45.235506 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":71,"threads_started":1}
I20260927 03:58:45.235803 22243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:45.228963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38318 (local address 127.15.185.126:43909)
0927 03:58:45.229014 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:45.229016 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:45.229143 (+ 127us) server_negotiation.cc:407] Connection header received
0927 03:58:45.229177 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:45.229178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:45.229188 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:45.229268 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:45.229544 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.230310 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.231040 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.231365 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.231435 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:45.232358 (+ 923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:45.232360 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:45.232360 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:45.232369 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:45.233981 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:45.234336 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:45.234337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:45.234338 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:45.234449 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:45.234777 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:45.234778 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:45.234779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:45.235121 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:45.235245 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:45.235519 (+ 274us) server_negotiation.cc:299] Negotiation successful
0927 03:58:45.235649 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 03:58:45.236119 22223 tablet_copy_service.cc:140] P 1868152744144c8f97d2a16fc4d34438: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 87685032f24e44c59ad7c48aaff26803 ({username='oryx', principal='oryx/127.15.185.125@KRBTEST.COM'} at 127.0.0.1:38318)
I20260927 03:58:45.236214 22223 tablet_copy_service.cc:161] P 1868152744144c8f97d2a16fc4d34438: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 87685032f24e44c59ad7c48aaff26803 at {username='oryx', principal='oryx/127.15.185.125@KRBTEST.COM'} at 127.0.0.1:38318: session id = 87685032f24e44c59ad7c48aaff26803-00000000000000000000000000000000
I20260927 03:58:45.237027 22223 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 1868152744144c8f97d2a16fc4d34438: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 03:58:45.237660 22667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 03:58:45.238894 22667 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803: tablet copy: Starting download of 0 data blocks...
I20260927 03:58:45.239066 22667 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803: tablet copy: Starting download of 1 WAL segments...
I20260927 03:58:45.240074 22667 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 03:58:45.241176 22223 tablet_copy_service.cc:342] P 1868152744144c8f97d2a16fc4d34438: Request end of tablet copy session 87685032f24e44c59ad7c48aaff26803-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.15.185.125@KRBTEST.COM'} at 127.0.0.1:38318
I20260927 03:58:45.241240 22223 tablet_copy_service.cc:433] P 1868152744144c8f97d2a16fc4d34438: ending tablet copy session 87685032f24e44c59ad7c48aaff26803-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 87685032f24e44c59ad7c48aaff26803
I20260927 03:58:45.243584 22667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:45.244803 22774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:45.244853 22775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:45.244912 22777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:45.245033 22667 server_base.cc:1034] running on GCE node
I20260927 03:58:45.245119 22667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:45.245245 22667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:45.246366 22667 hybrid_clock.cc:648] HybridClock initialized: now 1790481525246362 us; error 36 us; skew 500 ppm
Sep 27 03:58:45 dist-test-slave-glgz krb5kdc[22158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481525, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.185.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:45.247653 22667 init.cc:377] Logged in from keytab as oryx/127.15.185.125@KRBTEST.COM (short username oryx)
I20260927 03:58:45.247933 22667 webserver.cc:460] Webserver started at http://127.15.185.125:46837/ using document root <none> and password file <none>
I20260927 03:58:45.248076 22667 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:45.248123 22667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:45.248699 22667 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:58:45.249200 22784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:45.249321 22667 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:45.249367 22667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/master-1/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/master-1/wal
uuid: "87685032f24e44c59ad7c48aaff26803"
format_stamp: "Formatted at 2026-09-27 03:58:45 on dist-test-slave-glgz"
server_key: "b478e18f7041855f24486a118b358ec6"
server_key_iv: "4ee8c402259078a2a8ce8c6f196c0c00"
server_key_version: "encryptionkey@0"
I20260927 03:58:45.249423 22667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:45.262281 22667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:45.264000 22667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:45.267004 22667 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.125:36857
I20260927 03:58:45.267371 22836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.125:36857 every 8 connection(s)
I20260927 03:58:45.267406 22667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/master-1/data/info.pb
I20260927 03:58:45.268206 22837 sys_catalog.cc:263] Verifying existing consensus state
I20260927 03:58:45.268604 22837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803: Bootstrap starting.
I20260927 03:58:45.270757 22837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 03:58:45.270990 22837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803: Bootstrap complete.
I20260927 03:58:45.271251 22837 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1868152744144c8f97d2a16fc4d34438" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 43909 } } peers { permanent_uuid: "87685032f24e44c59ad7c48aaff26803" member_type: NON_VOTER last_known_addr { host: "127.15.185.125" port: 36857 } attrs { promote: true } }
I20260927 03:58:45.271355 22837 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 87685032f24e44c59ad7c48aaff26803, State: Initialized, Role: LEARNER
I20260927 03:58:45.271406 22837 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1868152744144c8f97d2a16fc4d34438" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 43909 } } peers { permanent_uuid: "87685032f24e44c59ad7c48aaff26803" member_type: NON_VOTER last_known_addr { host: "127.15.185.125" port: 36857 } attrs { promote: true } }
I20260927 03:58:45.271570 22837 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:58:45.271559 22840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1868152744144c8f97d2a16fc4d34438" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 43909 } } peers { permanent_uuid: "87685032f24e44c59ad7c48aaff26803" member_type: NON_VOTER last_known_addr { host: "127.15.185.125" port: 36857 } attrs { promote: true } } }
I20260927 03:58:45.271637 22840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803 [sys.catalog]: This master's current role is: LEARNER
I20260927 03:58:45.272962 22851 catalog_manager.cc:1259] Loaded cluster ID: 019b672ca57e471d9afa3c19afb55a10
I20260927 03:58:45.273130 22851 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803: loading cluster ID for follower catalog manager: success
I20260927 03:58:45.274353 22851 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803: acquiring CA information for follower catalog manager: success
I20260927 03:58:45.274677 22851 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 03:58:45.281544 22852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:45.274152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50610 (local address 127.15.185.125:36857)
0927 03:58:45.274288 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:45.274290 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:45.274301 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:58:45.274326 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:45.274327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:45.274353 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:45.275115 (+ 762us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:45.275253 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.275956 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.277020 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.277212 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.277266 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:45.278628 (+ 1362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:45.278629 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:45.278629 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:45.278635 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:45.280247 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:45.280701 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:45.280703 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:45.280703 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:45.280763 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:45.281014 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:45.281015 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:45.281015 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:45.281146 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:45.281249 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:45.281393 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 03:58:45.281464 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":47,"threads_started":1}
WARNING: no policy specified for oryx/127.15.185.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.185.124@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.185.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/krb5kdc/oryx_127.15.185.124.keytab.
Entry for principal oryx/127.15.185.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/krb5kdc/oryx_127.15.185.124.keytab.
I20260927 03:58:45.309857 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.124:46615
--webserver_interface=127.15.185.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/krb5kdc/oryx_127.15.185.124.keytab
--principal=oryx/127.15.185.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:45795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.185.126:43909,127.15.185.125:36857,127.15.185.124:46615
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:58:45.399004 22855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:45.399269 22855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:45.399291 22855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:45.400439 22855 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:58:45.400480 22855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:45.400493 22855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:45.400506 22855 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:58:45.400556 22855 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:58:45.401984 22855 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:45795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.15.185.126:43909,127.15.185.125:36857,127.15.185.124:46615
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/krb5kdc/oryx_127.15.185.124.keytab
--principal=oryx/127.15.185.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.124:46615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.15.185.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22855
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:45.402230 22855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:45.402455 22855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:45.407294 22861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:45.407424 22855 server_base.cc:1034] running on GCE node
W20260927 03:58:45.407306 22860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:45.407307 22863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:45.407745 22855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:45.408047 22855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:45.409238 22855 hybrid_clock.cc:648] HybridClock initialized: now 1790481525409224 us; error 45 us; skew 500 ppm
Sep 27 03:58:45 dist-test-slave-glgz krb5kdc[22158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481525, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.185.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:45.412031 22855 init.cc:377] Logged in from keytab as oryx/127.15.185.124@KRBTEST.COM (short username oryx)
I20260927 03:58:45.412429 22855 webserver.cc:460] Webserver started at http://127.15.185.124:44491/ using document root <none> and password file <none>
I20260927 03:58:45.412619 22855 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:45.412659 22855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:45.412747 22855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:45.413875 22855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/master-2/data/instance:
uuid: "08795fc9741d45d3953bb7183c4cb80f"
format_stamp: "Formatted at 2026-09-27 03:58:45 on dist-test-slave-glgz"
server_key: "ec140226dd04248307d069764f907385"
server_key_iv: "7f610693ca8d388c6c210b000da7cee9"
server_key_version: "encryptionkey@0"
I20260927 03:58:45.414181 22855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/master-2/wal/instance:
uuid: "08795fc9741d45d3953bb7183c4cb80f"
format_stamp: "Formatted at 2026-09-27 03:58:45 on dist-test-slave-glgz"
server_key: "ec140226dd04248307d069764f907385"
server_key_iv: "7f610693ca8d388c6c210b000da7cee9"
server_key_version: "encryptionkey@0"
I20260927 03:58:45.415604 22855 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:58:45.416363 22870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:45.416558 22855 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:45.416647 22855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/master-2/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/master-2/wal
uuid: "08795fc9741d45d3953bb7183c4cb80f"
format_stamp: "Formatted at 2026-09-27 03:58:45 on dist-test-slave-glgz"
server_key: "ec140226dd04248307d069764f907385"
server_key_iv: "7f610693ca8d388c6c210b000da7cee9"
server_key_version: "encryptionkey@0"
I20260927 03:58:45.416697 22855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:45.438105 22855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:45.440189 22855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:45.440317 22855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:45.444576 22855 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.124:46615
I20260927 03:58:45.444631 22922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.124:46615 every 8 connection(s)
I20260927 03:58:45.445066 22855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/master-2/data/info.pb
I20260927 03:58:45.446285 22923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:45.447958 22923 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.185.126" port: 43909 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 03:58:45 dist-test-slave-glgz krb5kdc[22158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481525, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.185.124@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM
I20260927 03:58:45.454984 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 22855
I20260927 03:58:45.457648 22243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:45.448734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38330 (local address 127.15.185.126:43909)
0927 03:58:45.448815 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:45.448818 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:45.449035 (+ 217us) server_negotiation.cc:407] Connection header received
0927 03:58:45.449626 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:45.449627 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:45.449641 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:45.449696 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:45.450380 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.450922 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.452066 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.452284 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.452385 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:45.454595 (+ 2210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:45.454596 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:45.454596 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:45.454603 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:45.456293 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:45.456727 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:45.456729 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:45.456729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:45.456797 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:45.457023 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:45.457024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:45.457024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:45.457164 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:45.457235 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:45.457459 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 03:58:45.457538 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 03:58:45.457743 22925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:45.448634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:43909 (local address 127.0.0.1:38330)
0927 03:58:45.448906 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:45.448944 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:45.449496 (+ 552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:45.449770 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:45.449773 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:45.450038 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:45.450233 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:45.450237 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.451020 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.451022 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:45.451945 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:45.451947 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.452405 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.452406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:45.452544 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:45.453086 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:45.453095 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:45.454511 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:45.456370 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:45.456375 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:45.456377 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:45.456641 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:45.456870 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:45.456871 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:45.456871 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:45.456960 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:45.457228 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:45.457231 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:45.457334 (+ 103us) client_negotiation.cc:769] Sending connection context
0927 03:58:45.457413 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 03:58:45.457519 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":97,"threads_started":1}
I20260927 03:58:45.458228 22923 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.185.125" port: 36857 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 03:58:45 dist-test-slave-glgz krb5kdc[22158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481524, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.185.124@KRBTEST.COM
Sep 27 03:58:45 dist-test-slave-glgz krb5kdc[22158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481525, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.185.124@KRBTEST.COM for oryx/127.15.185.125@KRBTEST.COM
I20260927 03:58:45.464780 22926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:45.455588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47618 (local address 127.15.185.124:46615)
0927 03:58:45.455762 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:45.455767 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:45.455794 (+ 27us) server_negotiation.cc:407] Connection header received
0927 03:58:45.455819 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:45.455821 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:45.455837 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:45.455933 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:45.456535 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.457513 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.458426 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.458812 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.458867 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:45.461553 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:45.461560 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:45.461560 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:45.461566 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:45.463326 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:45.463725 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:45.463729 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:45.463729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:45.463797 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:45.464031 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:45.464032 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:45.464032 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:45.464317 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:45.464515 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:45.464566 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 03:58:45.464644 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":90,"threads_started":1}
I20260927 03:58:45.466146 22925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:45.458486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.125:36857 (local address 127.0.0.1:50616)
0927 03:58:45.458536 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:45.458547 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:45.458582 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:45.458747 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:45.458748 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:45.458906 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:45.458956 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:45.458957 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.459772 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.459772 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:45.460590 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:45.460592 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.461085 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.461086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:45.461149 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:45.461551 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:45.461555 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:45.462977 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:45.465308 (+ 2331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:45.465308 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:45.465309 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:45.465487 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:45.465681 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:45.465682 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:45.465682 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:45.465727 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:45.465956 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:45.465957 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:45.465986 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 03:58:45.466030 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 03:58:45.466091 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 03:58:45.466178 22852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:45.458578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50616 (local address 127.15.185.125:36857)
0927 03:58:45.458633 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:45.458635 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:45.458642 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:58:45.458664 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:45.458664 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:45.458677 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:45.458744 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:45.459088 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.459672 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.460669 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.460990 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.461079 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:45.463093 (+ 2014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:45.463094 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:45.463094 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:45.463101 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:45.464983 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:45.465552 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:45.465553 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:45.465553 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:45.465614 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:45.465765 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:45.465765 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:45.465765 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:45.465907 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:45.465965 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:45.466050 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 03:58:45.466116 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 03:58:45.466583 22923 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.185.124" port: 46615 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 03:58:45 dist-test-slave-glgz krb5kdc[22158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481525, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.185.124@KRBTEST.COM for oryx/127.15.185.124@KRBTEST.COM
I20260927 03:58:45.472399 22925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:45.466888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.124:46615 (local address 127.0.0.1:47624)
0927 03:58:45.466938 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:45.466948 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:45.466990 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:45.467133 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:45.467136 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:45.467246 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:45.467271 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:45.467272 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.467844 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.467845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:45.468285 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:45.468286 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.468640 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.468640 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:45.468674 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:45.469057 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:45.469061 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:45.470090 (+ 1029us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:45.471480 (+ 1390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:45.471480 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:45.471481 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:45.471648 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:45.471858 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:45.471859 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:45.471859 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:45.471940 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:45.472170 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:45.472171 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:45.472203 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 03:58:45.472255 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 03:58:45.472342 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 03:58:45.472532 22926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:45.466941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47624 (local address 127.15.185.124:46615)
0927 03:58:45.466992 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:45.466993 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:45.467000 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:58:45.467037 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:45.467038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:45.467079 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:45.467131 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:45.467326 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.467760 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.468375 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.468571 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.468647 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:45.470161 (+ 1514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:45.470162 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:45.470162 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:45.470168 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:45.471381 (+ 1213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:45.471715 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:45.471715 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:45.471716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:45.471780 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:45.472014 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:45.472015 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:45.472015 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:45.472115 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:45.472166 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:45.472272 (+ 106us) server_negotiation.cc:299] Negotiation successful
0927 03:58:45.472384 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":36}
I20260927 03:58:45.473892 22923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f: Bootstrap starting.
I20260927 03:58:45.474864 22923 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:45.475204 22923 log.cc:824] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f: Log is configured to *not* fsync() on all Append() calls
I20260927 03:58:45.475957 22923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f: No bootstrap required, opened a new log
I20260927 03:58:45.477388 22923 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1868152744144c8f97d2a16fc4d34438" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 43909 } } peers { permanent_uuid: "87685032f24e44c59ad7c48aaff26803" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 36857 } } peers { permanent_uuid: "08795fc9741d45d3953bb7183c4cb80f" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 46615 } }
I20260927 03:58:45.477514 22923 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:45.477593 22923 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08795fc9741d45d3953bb7183c4cb80f, State: Initialized, Role: FOLLOWER
I20260927 03:58:45.477752 22923 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1868152744144c8f97d2a16fc4d34438" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 43909 } } peers { permanent_uuid: "87685032f24e44c59ad7c48aaff26803" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 36857 } } peers { permanent_uuid: "08795fc9741d45d3953bb7183c4cb80f" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 46615 } }
I20260927 03:58:45.478035 22928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1868152744144c8f97d2a16fc4d34438" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 43909 } } peers { permanent_uuid: "87685032f24e44c59ad7c48aaff26803" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 36857 } } peers { permanent_uuid: "08795fc9741d45d3953bb7183c4cb80f" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 46615 } } }
I20260927 03:58:45.478077 22923 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:58:45.478119 22928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f [sys.catalog]: This master's current role is: FOLLOWER
W20260927 03:58:45.479836 22939 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:58:45.479903 22939 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:58:45.482363 22855 master_runner.cc:428] Detected that this master 08795fc9741d45d3953bb7183c4cb80f is joining an existing cluster
I20260927 03:58:45.482403 22855 master_runner.cc:432] Initiating AddMaster RPC to add 127.15.185.124:46615
I20260927 03:58:45.491003 22926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:45.484238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47626 (local address 127.15.185.124:46615)
0927 03:58:45.484273 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:45.484275 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:45.484364 (+ 89us) server_negotiation.cc:407] Connection header received
0927 03:58:45.484455 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:45.484456 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:45.484464 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:45.484489 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:45.484910 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.485596 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.486610 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.486916 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.486956 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:45.488529 (+ 1573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:45.488530 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:45.488531 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:45.488536 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:45.489914 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:45.490315 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:45.490315 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:45.490316 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:45.490363 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:45.490579 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:45.490579 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:45.490580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:45.490668 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:45.490715 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:45.490881 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 03:58:45.490942 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":19}
I20260927 03:58:45.491183 22947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:45.483899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.124:46615 (local address 127.0.0.1:47626)
0927 03:58:45.484315 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:45.484323 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:45.484394 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:45.484540 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:45.484540 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:45.484752 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:45.484816 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:45.484818 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.485683 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.485684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:45.486519 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:45.486522 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.487024 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.487025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:45.487115 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:45.488010 (+ 895us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:45.488014 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:45.488441 (+ 427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:45.490011 (+ 1570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:45.490012 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:45.490013 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:45.490241 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:45.490422 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:45.490423 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:45.490423 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:45.490503 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:45.490731 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:45.490732 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:45.490777 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 03:58:45.490866 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 03:58:45.491009 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"mutex_wait_us":44,"thread_start_us":81,"threads_started":1}
I20260927 03:58:45.507604 22852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:45.483902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50624 (local address 127.15.185.125:36857)
0927 03:58:45.484067 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:45.484069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:45.484077 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:58:45.484102 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:45.484103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:45.484112 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:45.484194 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:45.484545 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.485304 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.486538 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.486832 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.486869 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:45.504711 (+ 17842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:45.504713 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:45.504714 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:45.504724 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:45.506360 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:45.506747 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:45.506748 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:45.506749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:45.506810 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:45.507013 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:45.507014 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:45.507015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:45.507202 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:45.507272 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:45.507420 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 03:58:45.507506 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
I20260927 03:58:45.507633 22946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:45.483660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.125:36857 (local address 127.0.0.1:50624)
0927 03:58:45.483949 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:45.483959 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:45.483995 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:45.484196 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:45.484197 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:45.484404 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:45.484460 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:45.484461 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.485402 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.485403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:45.486427 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:45.486429 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.486958 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.486959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:45.487028 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:45.503832 (+ 16804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:45.503840 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:45.504610 (+ 770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:45.506472 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:45.506474 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:45.506474 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:45.506659 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:45.506890 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:45.506891 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:45.506891 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:45.506956 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:45.507262 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:45.507263 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:45.507309 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 03:58:45.507396 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 03:58:45.507506 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":37,"thread_start_us":76,"threads_started":1}
I20260927 03:58:45.507828 22243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:45.483851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38336 (local address 127.15.185.126:43909)
0927 03:58:45.483944 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:45.483947 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:45.483954 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:58:45.483981 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:45.483982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:45.483993 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:45.484043 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:45.484538 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.485306 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.486528 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.486822 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.486865 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:45.505026 (+ 18161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:45.505027 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:45.505028 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:45.505037 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:45.506611 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:45.506955 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:45.506956 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:45.506957 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:45.507008 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:45.507228 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:45.507230 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:45.507230 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:45.507364 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:45.507432 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:45.507628 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 03:58:45.507719 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 03:58:45.508134 22945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:45.483644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:43909 (local address 127.0.0.1:38336)
0927 03:58:45.483789 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:45.483802 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:45.483854 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:45.484097 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:45.484097 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:45.484357 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:45.484436 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:45.484438 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.485402 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.485403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:45.486427 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:45.486429 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.486960 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.486960 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:45.487028 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:45.503832 (+ 16804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:45.503840 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:45.504583 (+ 743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:45.506705 (+ 2122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:45.506706 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:45.506706 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:45.506896 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:45.507099 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:45.507099 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:45.507099 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:45.507150 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:45.507463 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:45.507464 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:45.507513 (+ 49us) client_negotiation.cc:769] Sending connection context
0927 03:58:45.507908 (+ 395us) client_negotiation.cc:240] Negotiation successful
0927 03:58:45.508027 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":101,"thread_start_us":97,"threads_started":1}
Sep 27 03:58:45 dist-test-slave-glgz krb5kdc[22158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481524, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.185.126@KRBTEST.COM for oryx/127.15.185.124@KRBTEST.COM
I20260927 03:58:45.516296 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 22265
I20260927 03:58:45.517782 22926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:45.510379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47640 (local address 127.15.185.124:46615)
0927 03:58:45.510421 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:45.510423 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:45.510431 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:58:45.510487 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:45.510488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:45.510497 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:45.510545 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:45.510822 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.511368 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.512055 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.512257 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.512334 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:45.514260 (+ 1926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:45.514261 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:45.514262 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:45.514270 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:45.516055 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:45.516880 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:45.516881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:45.516882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:45.516947 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:45.517177 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:45.517178 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:45.517179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:45.517322 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:45.517398 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:45.517602 (+ 204us) server_negotiation.cc:299] Negotiation successful
0927 03:58:45.517682 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 03:58:45.517870 22761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:45.510314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.124:46615 (local address 127.0.0.1:47640)
0927 03:58:45.510369 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:45.510379 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:45.510420 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:45.510546 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:45.510547 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:45.510702 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:45.510742 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:45.510744 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.511453 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.511454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:45.511970 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:45.511971 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.512330 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.512331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:45.512395 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:45.512792 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:45.512798 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:45.514160 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:45.516151 (+ 1991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:45.516152 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:45.516152 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:45.516437 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:45.517028 (+ 591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:45.517029 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:45.517030 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:45.517100 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:45.517427 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:45.517428 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:45.517486 (+ 58us) client_negotiation.cc:769] Sending connection context
0927 03:58:45.517569 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 03:58:45.517697 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 03:58:45.518677 22203 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.15.185.124:46615: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "08795fc9741d45d3953bb7183c4cb80f"
member_type: NON_VOTER
last_known_addr {
host: "127.15.185.124"
port: 46615
}
attrs {
promote: true
}
}
dest_uuid: "1868152744144c8f97d2a16fc4d34438"
cas_config_opid_index: 5
I20260927 03:58:45.518950 22203 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1868152744144c8f97d2a16fc4d34438 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1868152744144c8f97d2a16fc4d34438" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 43909 } } peers { permanent_uuid: "87685032f24e44c59ad7c48aaff26803" member_type: NON_VOTER last_known_addr { host: "127.15.185.125" port: 36857 } attrs { promote: true } } peers { permanent_uuid: "08795fc9741d45d3953bb7183c4cb80f" member_type: NON_VOTER last_known_addr { host: "127.15.185.124" port: 46615 } attrs { promote: true } }
I20260927 03:58:45.519563 22762 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1868152744144c8f97d2a16fc4d34438 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 08795fc9741d45d3953bb7183c4cb80f (127.15.185.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1868152744144c8f97d2a16fc4d34438" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 43909 } } peers { permanent_uuid: "87685032f24e44c59ad7c48aaff26803" member_type: NON_VOTER last_known_addr { host: "127.15.185.125" port: 36857 } attrs { promote: true } } peers { permanent_uuid: "08795fc9741d45d3953bb7183c4cb80f" member_type: NON_VOTER last_known_addr { host: "127.15.185.124" port: 46615 } attrs { promote: true } } }
I20260927 03:58:45.520808 22762 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.15.185.124:46615
I20260927 03:58:45.520902 22762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1868152744144c8f97d2a16fc4d34438 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1868152744144c8f97d2a16fc4d34438" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1868152744144c8f97d2a16fc4d34438" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 43909 } } peers { permanent_uuid: "87685032f24e44c59ad7c48aaff26803" member_type: NON_VOTER last_known_addr { host: "127.15.185.125" port: 36857 } attrs { promote: true } } peers { permanent_uuid: "08795fc9741d45d3953bb7183c4cb80f" member_type: NON_VOTER last_known_addr { host: "127.15.185.124" port: 46615 } attrs { promote: true } } }
I20260927 03:58:45.521003 22762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1868152744144c8f97d2a16fc4d34438 [sys.catalog]: This master's current role is: LEADER
I20260927 03:58:45.521536 22897 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:45.522436 22897 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f [term 1 FOLLOWER]: Refusing update from remote peer 1868152744144c8f97d2a16fc4d34438: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 03:58:45.522689 22763 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1868152744144c8f97d2a16fc4d34438 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08795fc9741d45d3953bb7183c4cb80f" member_type: NON_VOTER last_known_addr { host: "127.15.185.124" port: 46615 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:58:45.524062 22897 raft_consensus.cc:858] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1868152744144c8f97d2a16fc4d34438" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 43909 } } peers { permanent_uuid: "87685032f24e44c59ad7c48aaff26803" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 36857 } } peers { permanent_uuid: "08795fc9741d45d3953bb7183c4cb80f" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 46615 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1868152744144c8f97d2a16fc4d34438" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 43909 } } peers { permanent_uuid: "87685032f24e44c59ad7c48aaff26803" member_type: NON_VOTER last_known_addr { host: "127.15.185.125" port: 36857 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1868152744144c8f97d2a16fc4d34438" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 43909 } } peers { permanent_uuid: "87685032f24e44c59ad7c48aaff26803" member_type: NON_VOTER last_known_addr { host: "127.15.185.125" port: 36857 } attrs { promote: true } } peers { permanent_uuid: "08795fc9741d45d3953bb7183c4cb80f" member_type: NON_VOTER last_known_addr { host: "127.15.185.124" port: 46615 } attrs { promote: true } }
W20260927 03:58:45.524142 22897 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1868152744144c8f97d2a16fc4d34438" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 43909 } } peers { permanent_uuid: "87685032f24e44c59ad7c48aaff26803" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 36857 } } peers { permanent_uuid: "08795fc9741d45d3953bb7183c4cb80f" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 46615 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1868152744144c8f97d2a16fc4d34438" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 43909 } } peers { permanent_uuid: "87685032f24e44c59ad7c48aaff26803" member_type: NON_VOTER last_known_addr { host: "127.15.185.125" port: 36857 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1868152744144c8f97d2a16fc4d34438" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 43909 } } peers { permanent_uuid: "87685032f24e44c59ad7c48aaff26803" member_type: NON_VOTER last_known_addr { host: "127.15.185.125" port: 36857 } attrs { promote: true } } peers { permanent_uuid: "08795fc9741d45d3953bb7183c4cb80f" member_type: NON_VOTER last_known_addr { host: "127.15.185.124" port: 46615 } attrs { promote: true } }
I20260927 03:58:45.524559 22855 master.cc:537] Master@127.15.185.124:46615 shutting down...
I20260927 03:58:45.524624 22897 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 87685032f24e44c59ad7c48aaff26803 (127.15.185.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1868152744144c8f97d2a16fc4d34438" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 43909 } } peers { permanent_uuid: "87685032f24e44c59ad7c48aaff26803" member_type: NON_VOTER last_known_addr { host: "127.15.185.125" port: 36857 } attrs { promote: true } } }
I20260927 03:58:45.525688 22928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1868152744144c8f97d2a16fc4d34438. Latest consensus state: current_term: 1 leader_uuid: "1868152744144c8f97d2a16fc4d34438" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1868152744144c8f97d2a16fc4d34438" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 43909 } } peers { permanent_uuid: "87685032f24e44c59ad7c48aaff26803" member_type: NON_VOTER last_known_addr { host: "127.15.185.125" port: 36857 } attrs { promote: true } } }
I20260927 03:58:45.525763 22928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260927 03:58:45.525802 22928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1868152744144c8f97d2a16fc4d34438" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1868152744144c8f97d2a16fc4d34438" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 43909 } } peers { permanent_uuid: "87685032f24e44c59ad7c48aaff26803" member_type: NON_VOTER last_known_addr { host: "127.15.185.125" port: 36857 } attrs { promote: true } } }
I20260927 03:58:45.525851 22928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260927 03:58:45.525884 22928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1868152744144c8f97d2a16fc4d34438" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1868152744144c8f97d2a16fc4d34438" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 43909 } } peers { permanent_uuid: "87685032f24e44c59ad7c48aaff26803" member_type: NON_VOTER last_known_addr { host: "127.15.185.125" port: 36857 } attrs { promote: true } } }
I20260927 03:58:45.525924 22928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260927 03:58:45.526024 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.65:35459
--local_ip_for_outbound_sockets=127.15.185.65
--tserver_master_addrs=127.15.185.126:43909
--webserver_port=39655
--webserver_interface=127.15.185.65
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/krb5kdc/oryx_127.15.185.65.keytab
--principal=oryx/127.15.185.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:45795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.15.185.126:43909,127.15.185.125:36857
--tserver_master_addrs=127.15.185.126:43909,127.15.185.125:36857,127.15.185.124:46615 with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:58:45.526713 22761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:45.519475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.125:36857 (local address 127.0.0.1:50628)
0927 03:58:45.521418 (+ 1943us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:45.521430 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:45.521475 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:45.521694 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:45.521695 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:45.521696 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:58:45.521742 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:45.521743 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.522778 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.522779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:45.524270 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:45.524274 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.526387 (+ 2113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.526388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:45.526474 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:45.526504 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 03:58:45.526564 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 03:58:45.526614 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1918}
I20260927 03:58:45.527020 22852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:45.521132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50628 (local address 127.15.185.125:36857)
0927 03:58:45.521367 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:45.521369 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:45.521529 (+ 160us) server_negotiation.cc:407] Connection header received
0927 03:58:45.521555 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:45.521556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:45.521580 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:45.521636 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:58:45.521811 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.522679 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.524410 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.525131 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.525213 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:45.525287 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:45.526868 (+ 1581us) server_negotiation.cc:299] Negotiation successful
0927 03:58:45.526918 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209}
I20260927 03:58:45.527268 22811 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803 [term 1 LEARNER]: Refusing update from remote peer 1868152744144c8f97d2a16fc4d34438: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 03:58:45.527504 22763 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1868152744144c8f97d2a16fc4d34438 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87685032f24e44c59ad7c48aaff26803" member_type: NON_VOTER last_known_addr { host: "127.15.185.125" port: 36857 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 03:58:45.528083 22811 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 08795fc9741d45d3953bb7183c4cb80f (127.15.185.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1868152744144c8f97d2a16fc4d34438" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 43909 } } peers { permanent_uuid: "87685032f24e44c59ad7c48aaff26803" member_type: NON_VOTER last_known_addr { host: "127.15.185.125" port: 36857 } attrs { promote: true } } peers { permanent_uuid: "08795fc9741d45d3953bb7183c4cb80f" member_type: NON_VOTER last_known_addr { host: "127.15.185.124" port: 46615 } attrs { promote: true } } }
I20260927 03:58:45.528239 22763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1868152744144c8f97d2a16fc4d34438 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1868152744144c8f97d2a16fc4d34438" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1868152744144c8f97d2a16fc4d34438" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 43909 } } peers { permanent_uuid: "87685032f24e44c59ad7c48aaff26803" member_type: NON_VOTER last_known_addr { host: "127.15.185.125" port: 36857 } attrs { promote: true } } peers { permanent_uuid: "08795fc9741d45d3953bb7183c4cb80f" member_type: NON_VOTER last_known_addr { host: "127.15.185.124" port: 46615 } attrs { promote: true } } }
I20260927 03:58:45.528328 22763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1868152744144c8f97d2a16fc4d34438 [sys.catalog]: This master's current role is: LEADER
I20260927 03:58:45.528692 22948 mvcc.cc:205] Tried to move back new op lower bound from 7333812324201570304 to 7333812324107886592. Current Snapshot: MvccSnapshot[applied={T|T < 7333812324119719936}]
W20260927 03:58:45.528923 22189 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1868152744144c8f97d2a16fc4d34438 -> Peer 08795fc9741d45d3953bb7183c4cb80f (127.15.185.124:46615): Couldn't send request to peer 08795fc9741d45d3953bb7183c4cb80f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 03:58:45.529362 22957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1868152744144c8f97d2a16fc4d34438" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1868152744144c8f97d2a16fc4d34438" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 43909 } } peers { permanent_uuid: "87685032f24e44c59ad7c48aaff26803" member_type: NON_VOTER last_known_addr { host: "127.15.185.125" port: 36857 } attrs { promote: true } } peers { permanent_uuid: "08795fc9741d45d3953bb7183c4cb80f" member_type: NON_VOTER last_known_addr { host: "127.15.185.124" port: 46615 } attrs { promote: true } } }
I20260927 03:58:45.529537 22957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803 [sys.catalog]: This master's current role is: LEARNER
I20260927 03:58:45.529620 22840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1868152744144c8f97d2a16fc4d34438. Latest consensus state: current_term: 1 leader_uuid: "1868152744144c8f97d2a16fc4d34438" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1868152744144c8f97d2a16fc4d34438" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 43909 } } peers { permanent_uuid: "87685032f24e44c59ad7c48aaff26803" member_type: NON_VOTER last_known_addr { host: "127.15.185.125" port: 36857 } attrs { promote: true } } peers { permanent_uuid: "08795fc9741d45d3953bb7183c4cb80f" member_type: NON_VOTER last_known_addr { host: "127.15.185.124" port: 46615 } attrs { promote: true } } }
I20260927 03:58:45.529748 22840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803 [sys.catalog]: This master's current role is: LEARNER
I20260927 03:58:45.531136 22762 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 1868152744144c8f97d2a16fc4d34438: attempting to promote NON_VOTER 87685032f24e44c59ad7c48aaff26803 to VOTER
I20260927 03:58:45.531353 22762 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1868152744144c8f97d2a16fc4d34438 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1868152744144c8f97d2a16fc4d34438" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 43909 } } peers { permanent_uuid: "87685032f24e44c59ad7c48aaff26803" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 36857 } attrs { promote: false } } peers { permanent_uuid: "08795fc9741d45d3953bb7183c4cb80f" member_type: NON_VOTER last_known_addr { host: "127.15.185.124" port: 46615 } attrs { promote: true } }
I20260927 03:58:45.532299 22811 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803 [term 1 LEARNER]: Refusing update from remote peer 1868152744144c8f97d2a16fc4d34438: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20260927 03:58:45.532317 22189 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1868152744144c8f97d2a16fc4d34438 -> Peer 08795fc9741d45d3953bb7183c4cb80f (127.15.185.124:46615): Couldn't send request to peer 08795fc9741d45d3953bb7183c4cb80f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 03:58:45.532557 22855 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260927 03:58:45.532655 22855 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260927 03:58:45.532616 22763 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1868152744144c8f97d2a16fc4d34438 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87685032f24e44c59ad7c48aaff26803" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 36857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 03:58:45.532686 22855 tablet_replica.cc:330] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f: stopping tablet replica
I20260927 03:58:45.533275 22763 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1868152744144c8f97d2a16fc4d34438 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 87685032f24e44c59ad7c48aaff26803 (127.15.185.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1868152744144c8f97d2a16fc4d34438" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 43909 } } peers { permanent_uuid: "87685032f24e44c59ad7c48aaff26803" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 36857 } attrs { promote: false } } peers { permanent_uuid: "08795fc9741d45d3953bb7183c4cb80f" member_type: NON_VOTER last_known_addr { host: "127.15.185.124" port: 46615 } attrs { promote: true } } }
I20260927 03:58:45.534385 22763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1868152744144c8f97d2a16fc4d34438 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1868152744144c8f97d2a16fc4d34438" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1868152744144c8f97d2a16fc4d34438" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 43909 } } peers { permanent_uuid: "87685032f24e44c59ad7c48aaff26803" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 36857 } attrs { promote: false } } peers { permanent_uuid: "08795fc9741d45d3953bb7183c4cb80f" member_type: NON_VOTER last_known_addr { host: "127.15.185.124" port: 46615 } attrs { promote: true } } }
I20260927 03:58:45.534490 22763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1868152744144c8f97d2a16fc4d34438 [sys.catalog]: This master's current role is: LEADER
I20260927 03:58:45.534554 22763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1868152744144c8f97d2a16fc4d34438 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1868152744144c8f97d2a16fc4d34438" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1868152744144c8f97d2a16fc4d34438" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 43909 } } peers { permanent_uuid: "87685032f24e44c59ad7c48aaff26803" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 36857 } attrs { promote: false } } peers { permanent_uuid: "08795fc9741d45d3953bb7183c4cb80f" member_type: NON_VOTER last_known_addr { host: "127.15.185.124" port: 46615 } attrs { promote: true } } }
I20260927 03:58:45.534636 22763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1868152744144c8f97d2a16fc4d34438 [sys.catalog]: This master's current role is: LEADER
I20260927 03:58:45.534941 22811 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 87685032f24e44c59ad7c48aaff26803 (127.15.185.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1868152744144c8f97d2a16fc4d34438" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 43909 } } peers { permanent_uuid: "87685032f24e44c59ad7c48aaff26803" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 36857 } attrs { promote: false } } peers { permanent_uuid: "08795fc9741d45d3953bb7183c4cb80f" member_type: NON_VOTER last_known_addr { host: "127.15.185.124" port: 46615 } attrs { promote: true } } }
I20260927 03:58:45.536059 22840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1868152744144c8f97d2a16fc4d34438" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1868152744144c8f97d2a16fc4d34438" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 43909 } } peers { permanent_uuid: "87685032f24e44c59ad7c48aaff26803" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 36857 } attrs { promote: false } } peers { permanent_uuid: "08795fc9741d45d3953bb7183c4cb80f" member_type: NON_VOTER last_known_addr { host: "127.15.185.124" port: 46615 } attrs { promote: true } } }
I20260927 03:58:45.536178 22840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:58:45.551687 22855 master.cc:552] Master@127.15.185.124:46615 shutdown complete.
I20260927 03:58:45.551754 22855 master_runner.cc:305] Clearing existing system tablet
I20260927 03:58:45.551899 22855 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:45.551976 22855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:45.552809 22855 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 03:58:45.553435 22960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:45.553551 22855 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:45.553624 22855 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:58:45.553653 22855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/master-2/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/master-2/wal
uuid: "08795fc9741d45d3953bb7183c4cb80f"
format_stamp: "Formatted at 2026-09-27 03:58:45 on dist-test-slave-glgz"
server_key: "ec140226dd04248307d069764f907385"
server_key_iv: "7f610693ca8d388c6c210b000da7cee9"
server_key_version: "encryptionkey@0"
I20260927 03:58:45.554464 22855 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:58:45.556144 22855 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 03:58:45.556229 22855 log.cc:1196] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f: Deleting WAL directory at /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260927 03:58:45.556488 22855 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f: Deleting consensus metadata
I20260927 03:58:45.556617 22855 master_runner.cc:315] Copying system tablet from 127.15.185.126:43909
I20260927 03:58:45.557281 22855 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f: tablet copy: Beginning tablet copy session from remote peer at address 127.15.185.126:43909
I20260927 03:58:45.567170 22966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:45.559368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:43909 (local address 127.0.0.1:38342)
0927 03:58:45.559531 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:45.559543 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:45.559625 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:45.559797 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:45.559798 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:45.560010 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:45.560073 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:45.560075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.561181 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.561182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:45.562136 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:45.562139 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.562643 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.562644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:45.562718 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:45.563291 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:45.563297 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:45.563740 (+ 443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:45.565842 (+ 2102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:45.565843 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:45.565843 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:45.566079 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:45.566328 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:45.566329 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:45.566329 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:45.566412 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:45.566730 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:45.566731 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:45.566783 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 03:58:45.566920 (+ 137us) client_negotiation.cc:240] Negotiation successful
0927 03:58:45.567032 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":65,"threads_started":1}
I20260927 03:58:45.567170 22243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:45.559422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38342 (local address 127.15.185.126:43909)
0927 03:58:45.559474 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:45.559476 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:45.559633 (+ 157us) server_negotiation.cc:407] Connection header received
0927 03:58:45.559695 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:45.559697 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:45.559709 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:45.559771 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:45.560168 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.561071 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.562228 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.562552 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.562637 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:45.563847 (+ 1210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:45.563848 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:45.563849 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:45.563857 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:45.565733 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:45.566172 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:45.566173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:45.566174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:45.566241 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:45.566497 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:45.566497 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:45.566498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:45.566647 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:45.566730 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:45.566960 (+ 230us) server_negotiation.cc:299] Negotiation successful
0927 03:58:45.567072 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 03:58:45.567857 22223 tablet_copy_service.cc:140] P 1868152744144c8f97d2a16fc4d34438: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 08795fc9741d45d3953bb7183c4cb80f ({username='oryx', principal='oryx/127.15.185.124@KRBTEST.COM'} at 127.0.0.1:38342)
I20260927 03:58:45.567926 22223 tablet_copy_service.cc:161] P 1868152744144c8f97d2a16fc4d34438: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 08795fc9741d45d3953bb7183c4cb80f at {username='oryx', principal='oryx/127.15.185.124@KRBTEST.COM'} at 127.0.0.1:38342: session id = 08795fc9741d45d3953bb7183c4cb80f-00000000000000000000000000000000
I20260927 03:58:45.568558 22223 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 1868152744144c8f97d2a16fc4d34438: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 03:58:45.569073 22855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 03:58:45.570415 22855 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f: tablet copy: Starting download of 0 data blocks...
I20260927 03:58:45.570571 22855 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f: tablet copy: Starting download of 1 WAL segments...
I20260927 03:58:45.571444 22855 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 03:58:45.572465 22223 tablet_copy_service.cc:342] P 1868152744144c8f97d2a16fc4d34438: Request end of tablet copy session 08795fc9741d45d3953bb7183c4cb80f-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.15.185.124@KRBTEST.COM'} at 127.0.0.1:38342
I20260927 03:58:45.572516 22223 tablet_copy_service.cc:433] P 1868152744144c8f97d2a16fc4d34438: ending tablet copy session 08795fc9741d45d3953bb7183c4cb80f-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 08795fc9741d45d3953bb7183c4cb80f
I20260927 03:58:45.575534 22855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:45.576808 22967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:45.576894 22968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:45.580263 22855 server_base.cc:1034] running on GCE node
W20260927 03:58:45.580258 22970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:45.580487 22855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:45.580632 22855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:45.581763 22855 hybrid_clock.cc:648] HybridClock initialized: now 1790481525581757 us; error 63 us; skew 500 ppm
Sep 27 03:58:45 dist-test-slave-glgz krb5kdc[22158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481525, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.185.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:45.583534 22855 init.cc:377] Logged in from keytab as oryx/127.15.185.124@KRBTEST.COM (short username oryx)
I20260927 03:58:45.583812 22855 webserver.cc:460] Webserver started at http://127.15.185.124:42859/ using document root <none> and password file <none>
I20260927 03:58:45.583947 22855 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:45.583989 22855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:45.584815 22855 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:58:45.585455 22977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:45.585639 22855 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:45.585700 22855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/master-2/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/master-2/wal
uuid: "08795fc9741d45d3953bb7183c4cb80f"
format_stamp: "Formatted at 2026-09-27 03:58:45 on dist-test-slave-glgz"
server_key: "ec140226dd04248307d069764f907385"
server_key_iv: "7f610693ca8d388c6c210b000da7cee9"
server_key_version: "encryptionkey@0"
I20260927 03:58:45.585764 22855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:45.598973 22855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:45.601016 22855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:45.604256 22855 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.124:46615
I20260927 03:58:45.604648 22855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/master-2/data/info.pb
I20260927 03:58:45.607158 23029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.124:46615 every 8 connection(s)
I20260927 03:58:45.608099 23030 sys_catalog.cc:263] Verifying existing consensus state
I20260927 03:58:45.608687 23030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f: Bootstrap starting.
I20260927 03:58:45.611302 23030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 03:58:45.611562 23030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f: Bootstrap complete.
I20260927 03:58:45.611766 23030 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1868152744144c8f97d2a16fc4d34438" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 43909 } } peers { permanent_uuid: "87685032f24e44c59ad7c48aaff26803" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 36857 } attrs { promote: false } } peers { permanent_uuid: "08795fc9741d45d3953bb7183c4cb80f" member_type: NON_VOTER last_known_addr { host: "127.15.185.124" port: 46615 } attrs { promote: true } }
I20260927 03:58:45.611860 23030 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 08795fc9741d45d3953bb7183c4cb80f, State: Initialized, Role: LEARNER
I20260927 03:58:45.611907 23030 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1868152744144c8f97d2a16fc4d34438" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 43909 } } peers { permanent_uuid: "87685032f24e44c59ad7c48aaff26803" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 36857 } attrs { promote: false } } peers { permanent_uuid: "08795fc9741d45d3953bb7183c4cb80f" member_type: NON_VOTER last_known_addr { host: "127.15.185.124" port: 46615 } attrs { promote: true } }
I20260927 03:58:45.612169 23030 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:58:45.612200 23033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1868152744144c8f97d2a16fc4d34438" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 43909 } } peers { permanent_uuid: "87685032f24e44c59ad7c48aaff26803" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 36857 } attrs { promote: false } } peers { permanent_uuid: "08795fc9741d45d3953bb7183c4cb80f" member_type: NON_VOTER last_known_addr { host: "127.15.185.124" port: 46615 } attrs { promote: true } } }
I20260927 03:58:45.612479 23033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f [sys.catalog]: This master's current role is: LEARNER
I20260927 03:58:45.615137 23044 catalog_manager.cc:1259] Loaded cluster ID: 019b672ca57e471d9afa3c19afb55a10
I20260927 03:58:45.615262 23044 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f: loading cluster ID for follower catalog manager: success
I20260927 03:58:45.616699 23044 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f: acquiring CA information for follower catalog manager: success
I20260927 03:58:45.617035 23044 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 03:58:45.634275 22952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:45.634500 22952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:45.634526 22952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:45.635915 22952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:45.635964 22952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:45.635995 22952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.65
I20260927 03:58:45.637722 22952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:45795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/krb5kdc/oryx_127.15.185.65.keytab
--principal=oryx/127.15.185.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.65:35459
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.185.65
--webserver_port=39655
--tserver_master_addrs=127.15.185.126:43909,127.15.185.125:36857,127.15.185.124:46615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.65
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:45.638010 22952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:45.638290 22952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:45.644109 23052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:45.644156 23049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:45.644140 23050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:45.644857 22952 server_base.cc:1034] running on GCE node
I20260927 03:58:45.645080 22952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:45.645363 22952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:45.646562 22952 hybrid_clock.cc:648] HybridClock initialized: now 1790481525646543 us; error 26 us; skew 500 ppm
Sep 27 03:58:45 dist-test-slave-glgz krb5kdc[22158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481525, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.185.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:45.649467 22952 init.cc:377] Logged in from keytab as oryx/127.15.185.65@KRBTEST.COM (short username oryx)
I20260927 03:58:45.649991 22952 webserver.cc:460] Webserver started at http://127.15.185.65:39655/ using document root <none> and password file <none>
I20260927 03:58:45.650246 22952 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:45.650331 22952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:45.651804 22952 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260927 03:58:45.652478 23059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:45.652642 22952 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:58:45.652745 22952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-0/wal
uuid: "ce9010ea1dc94a0f91d6c8b10895c2f1"
format_stamp: "Formatted at 2026-09-27 03:58:44 on dist-test-slave-glgz"
server_key: "b31969b28bec3e81e8b243f2dcbbe371"
server_key_iv: "99b02b3fff23c358223e042d9a768da7"
server_key_version: "encryptionkey@0"
I20260927 03:58:45.653164 22952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:45.678618 22952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:45.681062 22952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:45.681233 22952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:45.681488 22952 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:58:45.681816 22952 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:58:45.681881 22952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:45.681922 22952 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:58:45.681950 22952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:45.688097 22952 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.65:35459
I20260927 03:58:45.688130 23172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.65:35459 every 8 connection(s)
I20260927 03:58:45.688482 22952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
I20260927 03:58:45.693992 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 22952
I20260927 03:58:45.694159 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 22399
Sep 27 03:58:45 dist-test-slave-glgz krb5kdc[22158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481525, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.185.65@KRBTEST.COM for oryx/127.15.185.125@KRBTEST.COM
Sep 27 03:58:45 dist-test-slave-glgz krb5kdc[22158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481525, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.185.65@KRBTEST.COM for oryx/127.15.185.124@KRBTEST.COM
Sep 27 03:58:45 dist-test-slave-glgz krb5kdc[22158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481525, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.185.65@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM
I20260927 03:58:45.702261 23178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:45.689291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:58217 (local address 127.15.185.124:46615)
0927 03:58:45.689676 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:45.689678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:45.689686 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:58:45.690727 (+ 1041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:45.690728 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:45.690745 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:45.690820 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:45.691790 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.692775 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.693545 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.693862 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.693927 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:45.698455 (+ 4528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:45.698457 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:45.698457 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:45.698465 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:45.700552 (+ 2087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:45.701097 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:45.701098 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:45.701099 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:45.701162 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:45.701444 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:45.701445 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:45.701445 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:45.701608 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:45.701653 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:45.702036 (+ 383us) server_negotiation.cc:299] Negotiation successful
0927 03:58:45.702141 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":59,"threads_started":1}
I20260927 03:58:45.702271 22852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:45.689747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:50907 (local address 127.15.185.125:36857)
0927 03:58:45.689792 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:45.689793 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:45.689822 (+ 29us) server_negotiation.cc:407] Connection header received
0927 03:58:45.690753 (+ 931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:45.690755 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:45.690767 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:45.690799 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:45.691776 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.692279 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.693623 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.693932 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.693997 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:45.698359 (+ 4362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:45.698361 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:45.698362 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:45.698370 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:45.700406 (+ 2036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:45.701116 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:45.701118 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:45.701118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:45.701196 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:45.701478 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:45.701479 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:45.701479 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:45.701641 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:45.701691 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:45.702050 (+ 359us) server_negotiation.cc:299] Negotiation successful
0927 03:58:45.702142 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 03:58:45.702363 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.66:40143
--local_ip_for_outbound_sockets=127.15.185.66
--tserver_master_addrs=127.15.185.126:43909
--webserver_port=45427
--webserver_interface=127.15.185.66
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/krb5kdc/oryx_127.15.185.66.keytab
--principal=oryx/127.15.185.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:45795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.15.185.126:43909,127.15.185.125:36857
--tserver_master_addrs=127.15.185.126:43909,127.15.185.125:36857,127.15.185.124:46615 with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:58:45.702816 22243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:45.689229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:53675 (local address 127.15.185.126:43909)
0927 03:58:45.689282 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:45.689284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:45.689544 (+ 260us) server_negotiation.cc:407] Connection header received
0927 03:58:45.690879 (+ 1335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:45.690880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:45.690891 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:45.690947 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:45.691788 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.692552 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.694054 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.694816 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.694886 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:45.698350 (+ 3464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:45.698352 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:45.698353 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:45.698386 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:45.701179 (+ 2793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:45.701620 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:45.701621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:45.701621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:45.701678 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:45.701886 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:45.701887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:45.701887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:45.702030 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:45.702091 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:45.702645 (+ 554us) server_negotiation.cc:299] Negotiation successful
0927 03:58:45.702734 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 03:58:45.702817 23179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:45.689236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.124:46615 (local address 127.15.185.65:58217)
0927 03:58:45.689489 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:45.689500 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:45.690611 (+ 1111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:45.690902 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:45.690905 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:45.691415 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:45.691674 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:45.691680 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.692851 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.692852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:45.693448 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:45.693450 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.693997 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.693998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:45.694154 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:45.694871 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:45.694877 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:45.698347 (+ 3470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:45.700644 (+ 2297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:45.700646 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:45.700646 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:45.701008 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:45.701230 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:45.701231 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:45.701231 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:45.701375 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:45.701703 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:45.701708 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:45.701885 (+ 177us) client_negotiation.cc:769] Sending connection context
0927 03:58:45.702336 (+ 451us) client_negotiation.cc:240] Negotiation successful
0927 03:58:45.702566 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":152,"thread_start_us":99,"threads_started":1}
I20260927 03:58:45.703029 23177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:45.689236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:43909 (local address 127.15.185.65:53675)
0927 03:58:45.689415 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:45.689457 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:45.690611 (+ 1154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:45.690964 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:45.690965 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:45.691410 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:45.691674 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:45.691680 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.692633 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.692634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:45.693549 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:45.693551 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.694942 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.694943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:45.695014 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:45.695430 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:45.695435 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:45.698218 (+ 2783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:45.701272 (+ 3054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:45.701273 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:45.701274 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:45.701537 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:45.701753 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:45.701754 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:45.701754 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:45.701812 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:45.702118 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:45.702119 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:45.702157 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 03:58:45.702236 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 03:58:45.702476 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":15,"thread_start_us":99,"threads_started":1}
I20260927 03:58:45.703346 23180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:45.689652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.125:36857 (local address 127.15.185.65:50907)
0927 03:58:45.689770 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:45.689779 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:45.690629 (+ 850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:45.690910 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:45.690911 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:45.691411 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:45.691674 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:45.691679 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.692389 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.692390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:45.693556 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:45.693557 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.694011 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.694012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:45.694157 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:45.694781 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:45.694791 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:45.698247 (+ 3456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:45.700514 (+ 2267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:45.700522 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:45.700525 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:45.701010 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:45.701278 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:45.701278 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:45.701279 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:45.701391 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:45.701746 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:45.701747 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:45.701898 (+ 151us) client_negotiation.cc:769] Sending connection context
0927 03:58:45.703080 (+ 1182us) client_negotiation.cc:240] Negotiation successful
0927 03:58:45.703191 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":41,"thread_start_us":85,"threads_started":1}
I20260927 03:58:45.703457 23174 heartbeater.cc:347] Connected to a master server at 127.15.185.126:43909
I20260927 03:58:45.703553 23174 heartbeater.cc:464] Registering TS with master...
I20260927 03:58:45.703758 23174 heartbeater.cc:511] Master 127.15.185.126:43909 requested a full tablet report, sending...
I20260927 03:58:45.703972 23173 heartbeater.cc:347] Connected to a master server at 127.15.185.124:46615
I20260927 03:58:45.704017 23173 heartbeater.cc:464] Registering TS with master...
I20260927 03:58:45.704077 23173 heartbeater.cc:511] Master 127.15.185.124:46615 requested a full tablet report, sending...
I20260927 03:58:45.704221 23175 heartbeater.cc:347] Connected to a master server at 127.15.185.125:36857
I20260927 03:58:45.704250 23175 heartbeater.cc:464] Registering TS with master...
I20260927 03:58:45.704245 22203 ts_manager.cc:194] Re-registered known tserver with Master: ce9010ea1dc94a0f91d6c8b10895c2f1 (127.15.185.65:35459)
I20260927 03:58:45.704303 23175 heartbeater.cc:511] Master 127.15.185.125:36857 requested a full tablet report, sending...
I20260927 03:58:45.704480 22994 ts_manager.cc:194] Registered new tserver with Master: ce9010ea1dc94a0f91d6c8b10895c2f1 (127.15.185.65:35459)
I20260927 03:58:45.704769 22203 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.185.65@KRBTEST.COM'} at 127.15.185.65:53675
I20260927 03:58:45.704905 22801 ts_manager.cc:194] Registered new tserver with Master: ce9010ea1dc94a0f91d6c8b10895c2f1 (127.15.185.65:35459)
W20260927 03:58:45.785816 23181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:45.786033 23181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:45.786057 23181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:45.787357 23181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:45.787398 23181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:45.787424 23181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.66
I20260927 03:58:45.788858 23181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:45795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/krb5kdc/oryx_127.15.185.66.keytab
--principal=oryx/127.15.185.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.66:40143
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.185.66
--webserver_port=45427
--tserver_master_addrs=127.15.185.126:43909,127.15.185.125:36857,127.15.185.124:46615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.66
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:45.789060 23181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:45.789232 23181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:45.794260 23187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:45.794282 23189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:45.794335 23181 server_base.cc:1034] running on GCE node
W20260927 03:58:45.794355 23186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:45.794721 23181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:45.794975 23181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:45.796146 23181 hybrid_clock.cc:648] HybridClock initialized: now 1790481525796125 us; error 37 us; skew 500 ppm
Sep 27 03:58:45 dist-test-slave-glgz krb5kdc[22158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481525, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.185.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:45.798898 23181 init.cc:377] Logged in from keytab as oryx/127.15.185.66@KRBTEST.COM (short username oryx)
I20260927 03:58:45.799404 23181 webserver.cc:460] Webserver started at http://127.15.185.66:45427/ using document root <none> and password file <none>
I20260927 03:58:45.799639 23181 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:45.799718 23181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:45.801084 23181 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:58:45.801781 23196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:45.801952 23181 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:58:45.802059 23181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-1/wal
uuid: "55d1a890d9f4473fa298231905647c4a"
format_stamp: "Formatted at 2026-09-27 03:58:44 on dist-test-slave-glgz"
server_key: "d091847a5c1b7506346785e1bd34bd3e"
server_key_iv: "5c42f956c8dd4d23e367becf85254f8a"
server_key_version: "encryptionkey@0"
I20260927 03:58:45.802399 23181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:45.818737 23181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:45.821175 23181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:45.821337 23181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:45.821583 23181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:58:45.821892 23181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:58:45.821950 23181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:45.821986 23181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:58:45.822013 23181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:45.829206 23181 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.66:40143
I20260927 03:58:45.829269 23309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.66:40143 every 8 connection(s)
I20260927 03:58:45.829775 23181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
Sep 27 03:58:45 dist-test-slave-glgz krb5kdc[22158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481525, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.185.66@KRBTEST.COM for oryx/127.15.185.125@KRBTEST.COM
Sep 27 03:58:45 dist-test-slave-glgz krb5kdc[22158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481525, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.185.66@KRBTEST.COM for oryx/127.15.185.124@KRBTEST.COM
Sep 27 03:58:45 dist-test-slave-glgz krb5kdc[22158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481525, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.185.66@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM
I20260927 03:58:45.838672 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 23181
I20260927 03:58:45.838828 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 22533
I20260927 03:58:45.843523 22243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:45.830817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:55213 (local address 127.15.185.126:43909)
0927 03:58:45.830894 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:45.830897 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:45.830908 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:58:45.832053 (+ 1145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:45.832054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:45.832067 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:45.832136 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:45.833213 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.833986 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.835104 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.835378 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.835435 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:45.838835 (+ 3400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:45.838836 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:45.838837 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:45.838845 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:45.841782 (+ 2937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:45.842282 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:45.842283 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:45.842283 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:45.842339 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:45.842662 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:45.842663 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:45.842663 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:45.842828 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:45.842883 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:45.843272 (+ 389us) server_negotiation.cc:299] Negotiation successful
0927 03:58:45.843365 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 03:58:45.844147 23314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:45.830526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:43909 (local address 127.15.185.66:55213)
0927 03:58:45.830746 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:45.830789 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:45.831917 (+ 1128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:45.832212 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:45.832229 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:45.832810 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:45.833099 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:45.833105 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.834089 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.834089 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:45.835001 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:45.835003 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.835486 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.835487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:45.835558 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:45.836290 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:45.836295 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:45.838712 (+ 2417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:45.841872 (+ 3160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:45.841880 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:45.841883 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:45.842199 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:45.842434 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:45.842435 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:45.842435 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:45.842586 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:45.842896 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:45.842901 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:45.843098 (+ 197us) client_negotiation.cc:769] Sending connection context
0927 03:58:45.843624 (+ 526us) client_negotiation.cc:240] Negotiation successful
0927 03:58:45.843859 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":50,"thread_start_us":84,"threads_started":1}
I20260927 03:58:45.844753 23311 heartbeater.cc:347] Connected to a master server at 127.15.185.126:43909
I20260927 03:58:45.844839 23178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:45.830626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:54149 (local address 127.15.185.124:46615)
0927 03:58:45.830672 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:45.830674 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:45.831214 (+ 540us) server_negotiation.cc:407] Connection header received
0927 03:58:45.832052 (+ 838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:45.832054 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:45.832070 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:45.832102 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:45.833194 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.833777 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.834873 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.835169 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.835229 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:45.838256 (+ 3027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:45.838257 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:45.838258 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:45.838266 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:45.840706 (+ 2440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:45.844031 (+ 3325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:45.844032 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:45.844032 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:45.844095 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:45.844312 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:45.844313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:45.844313 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:45.844434 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:45.844488 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:45.844658 (+ 170us) server_negotiation.cc:299] Negotiation successful
0927 03:58:45.844741 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 03:58:45.844883 23311 heartbeater.cc:464] Registering TS with master...
I20260927 03:58:45.845016 23316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:45.830526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.124:46615 (local address 127.15.185.66:54149)
0927 03:58:45.831151 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:45.831159 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:45.831929 (+ 770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:45.832218 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:45.832221 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:45.832817 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:45.833099 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:45.833103 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.833878 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.833879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:45.834784 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:45.834787 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.835288 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.835289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:45.835432 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:45.836168 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:45.836173 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:45.838141 (+ 1968us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:45.843671 (+ 5530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:45.843672 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:45.843673 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:45.843941 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:45.844159 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:45.844160 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:45.844161 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:45.844224 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:45.844527 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:45.844527 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:45.844559 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 03:58:45.844633 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 03:58:45.844733 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"mutex_wait_us":105,"thread_start_us":87,"threads_started":1}
I20260927 03:58:45.845081 23311 heartbeater.cc:511] Master 127.15.185.126:43909 requested a full tablet report, sending...
I20260927 03:58:45.845317 23310 heartbeater.cc:347] Connected to a master server at 127.15.185.124:46615
I20260927 03:58:45.845366 23310 heartbeater.cc:464] Registering TS with master...
I20260927 03:58:45.845439 23310 heartbeater.cc:511] Master 127.15.185.124:46615 requested a full tablet report, sending...
I20260927 03:58:45.845466 22203 ts_manager.cc:194] Re-registered known tserver with Master: 55d1a890d9f4473fa298231905647c4a (127.15.185.66:40143)
I20260927 03:58:45.845727 22994 ts_manager.cc:194] Registered new tserver with Master: 55d1a890d9f4473fa298231905647c4a (127.15.185.66:40143)
I20260927 03:58:45.846015 22203 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.185.66@KRBTEST.COM'} at 127.15.185.66:55213
I20260927 03:58:45.846402 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.67:33539
--local_ip_for_outbound_sockets=127.15.185.67
--tserver_master_addrs=127.15.185.126:43909
--webserver_port=45851
--webserver_interface=127.15.185.67
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/krb5kdc/oryx_127.15.185.67.keytab
--principal=oryx/127.15.185.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:45795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.15.185.126:43909,127.15.185.125:36857
--tserver_master_addrs=127.15.185.126:43909,127.15.185.125:36857,127.15.185.124:46615 with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:58:45.848112 22852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:45.830962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:53487 (local address 127.15.185.125:36857)
0927 03:58:45.831031 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:45.831033 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:45.831108 (+ 75us) server_negotiation.cc:407] Connection header received
0927 03:58:45.832188 (+ 1080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:45.832190 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:45.832200 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:45.832255 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:45.833212 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.833994 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.835140 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.835342 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.835438 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:45.837751 (+ 2313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:45.837752 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:45.837753 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:45.837759 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:45.840118 (+ 2359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:45.847360 (+ 7242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:45.847362 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:45.847362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:45.847441 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:45.847673 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:45.847673 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:45.847674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:45.847776 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:45.847828 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:45.847987 (+ 159us) server_negotiation.cc:299] Negotiation successful
0927 03:58:45.848054 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 03:58:45.848424 23315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:45.830526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.125:36857 (local address 127.15.185.66:53487)
0927 03:58:45.831034 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:45.831063 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:45.831918 (+ 855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:45.832278 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:45.832279 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:45.832796 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:45.833099 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:45.833105 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.834089 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.834089 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:45.835006 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:45.835008 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.835439 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.835440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:45.835507 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:45.836115 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:45.836124 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:45.837659 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:45.847118 (+ 9459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:45.847119 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:45.847120 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:45.847285 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:45.847521 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:45.847523 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:45.847523 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:45.847597 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:45.847834 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:45.847835 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:45.847878 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 03:58:45.847942 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 03:58:45.848066 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"mutex_wait_us":81,"thread_start_us":86,"threads_started":1}
I20260927 03:58:45.848798 23312 heartbeater.cc:347] Connected to a master server at 127.15.185.125:36857
I20260927 03:58:45.848851 23312 heartbeater.cc:464] Registering TS with master...
I20260927 03:58:45.848922 23312 heartbeater.cc:511] Master 127.15.185.125:36857 requested a full tablet report, sending...
I20260927 03:58:45.849211 22801 ts_manager.cc:194] Registered new tserver with Master: 55d1a890d9f4473fa298231905647c4a (127.15.185.66:40143)
W20260927 03:58:45.930294 23317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:45.930496 23317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:45.930519 23317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:45.931710 23317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:45.931757 23317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:45.931783 23317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.67
I20260927 03:58:45.933182 23317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:45795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/krb5kdc/oryx_127.15.185.67.keytab
--principal=oryx/127.15.185.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.67:33539
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.185.67
--webserver_port=45851
--tserver_master_addrs=127.15.185.126:43909,127.15.185.125:36857,127.15.185.124:46615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23317
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.67
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:45.933373 23317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:45.933545 23317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:45.938609 23322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:45.938620 23325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:45.938676 23323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:45.938835 23317 server_base.cc:1034] running on GCE node
I20260927 03:58:45.939149 23317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:45.939428 23317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:45.940570 23317 hybrid_clock.cc:648] HybridClock initialized: now 1790481525940560 us; error 27 us; skew 500 ppm
Sep 27 03:58:45 dist-test-slave-glgz krb5kdc[22158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481525, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.185.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:45.943308 23317 init.cc:377] Logged in from keytab as oryx/127.15.185.67@KRBTEST.COM (short username oryx)
I20260927 03:58:45.943756 23317 webserver.cc:460] Webserver started at http://127.15.185.67:45851/ using document root <none> and password file <none>
I20260927 03:58:45.943967 23317 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:45.944020 23317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:45.945345 23317 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:58:45.945858 23332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:45.946038 23317 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:45.946141 23317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-2/wal
uuid: "20ea15da3bd74f2498765c4f9733de16"
format_stamp: "Formatted at 2026-09-27 03:58:44 on dist-test-slave-glgz"
server_key: "6334a29d4bd30fdedca82594f71b52f5"
server_key_iv: "4f77e68d5323fdb41a7efa9b2b687c5d"
server_key_version: "encryptionkey@0"
I20260927 03:58:45.946483 23317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:45.961838 23317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:45.963977 23317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:45.964152 23317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:45.964397 23317 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:58:45.964717 23317 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:58:45.964773 23317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:45.964843 23317 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:58:45.964882 23317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:45.970846 23317 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.67:33539
I20260927 03:58:45.970892 23445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.67:33539 every 8 connection(s)
I20260927 03:58:45.971380 23317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
I20260927 03:58:45.971534 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 23317
Sep 27 03:58:45 dist-test-slave-glgz krb5kdc[22158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481525, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.185.67@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM
Sep 27 03:58:45 dist-test-slave-glgz krb5kdc[22158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481525, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.185.67@KRBTEST.COM for oryx/127.15.185.124@KRBTEST.COM
Sep 27 03:58:45 dist-test-slave-glgz krb5kdc[22158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481525, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.185.67@KRBTEST.COM for oryx/127.15.185.125@KRBTEST.COM
I20260927 03:58:45.983675 23459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:45.972962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47646 (local address 127.15.185.124:46615)
0927 03:58:45.973846 (+ 884us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:45.973848 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:45.973855 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:58:45.973877 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:45.973878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:45.973890 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:45.974169 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:45.974887 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.975699 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.976702 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.977005 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.977064 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:45.979309 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:45.979310 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:45.979311 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:45.979319 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:45.981761 (+ 2442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:45.982421 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:45.982422 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:45.982423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:45.982488 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:45.982699 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:45.982699 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:45.982700 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:45.982845 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:45.982948 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:45.983413 (+ 465us) server_negotiation.cc:299] Negotiation successful
0927 03:58:45.983549 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":847,"thread_start_us":56,"threads_started":1}
I20260927 03:58:45.984900 22243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:45.972547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:47583 (local address 127.15.185.126:43909)
0927 03:58:45.972636 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:45.972638 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:45.972703 (+ 65us) server_negotiation.cc:407] Connection header received
0927 03:58:45.973464 (+ 761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:45.973465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:45.973478 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:45.973511 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:45.974691 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.975482 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.977031 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.977345 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.977383 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:45.980101 (+ 2718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:45.980102 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:45.980103 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:45.980109 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:45.982215 (+ 2106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:45.983484 (+ 1269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:45.983485 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:45.983485 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:45.983544 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:45.984086 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:45.984087 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:45.984088 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:45.984228 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:45.984272 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:45.984684 (+ 412us) server_negotiation.cc:299] Negotiation successful
0927 03:58:45.984775 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 03:58:45.985726 23454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:45.972399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:43909 (local address 127.15.185.67:47583)
0927 03:58:45.972632 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:45.972644 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:45.973341 (+ 697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:45.973703 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:45.973708 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:45.974268 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:45.974580 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:45.974585 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.975624 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.975624 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:45.976940 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:45.976942 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.977462 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.977462 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:45.977603 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:45.978082 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:45.978092 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:45.979993 (+ 1901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:45.982336 (+ 2343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:45.982344 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:45.982346 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:45.982664 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:45.983618 (+ 954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:45.983619 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:45.983619 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:45.984005 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:45.984322 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:45.984328 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:45.984503 (+ 175us) client_negotiation.cc:769] Sending connection context
0927 03:58:45.985272 (+ 769us) client_negotiation.cc:240] Negotiation successful
0927 03:58:45.985465 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":30,"thread_start_us":67,"threads_started":1}
I20260927 03:58:45.986009 23178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:45.972455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:57477 (local address 127.15.185.124:46615)
0927 03:58:45.972600 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:45.972602 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:45.972687 (+ 85us) server_negotiation.cc:407] Connection header received
0927 03:58:45.973463 (+ 776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:45.973465 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:45.973478 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:45.973511 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:45.974690 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.975502 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.979136 (+ 3634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.979456 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.979502 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:45.982123 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:45.982125 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:45.982125 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:45.982134 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:45.983854 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:45.985162 (+ 1308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:45.985163 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:45.985163 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:45.985215 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:45.985447 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:45.985447 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:45.985448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:45.985570 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:45.985617 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:45.985824 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 03:58:45.985907 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260927 03:58:45.986280 23461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:45.973180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38348 (local address 127.15.185.126:43909)
0927 03:58:45.974248 (+ 1068us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:45.974250 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:45.974258 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:58:45.974278 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:45.974279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:45.974290 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:45.974574 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:45.975410 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.976136 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.979612 (+ 3476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.979931 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.980423 (+ 492us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:45.981746 (+ 1323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:45.981747 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:45.981748 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:45.981756 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:45.984142 (+ 2386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:45.984981 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:45.984981 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:45.984982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:45.985040 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:45.985394 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:45.985395 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:45.985395 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:45.985688 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:45.986064 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:45.986112 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 03:58:45.986191 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1042,"thread_start_us":77,"threads_started":1}
I20260927 03:58:45.986579 23447 heartbeater.cc:347] Connected to a master server at 127.15.185.126:43909
I20260927 03:58:45.986675 23447 heartbeater.cc:464] Registering TS with master...
I20260927 03:58:45.986873 23447 heartbeater.cc:511] Master 127.15.185.126:43909 requested a full tablet report, sending...
I20260927 03:58:45.987269 22203 ts_manager.cc:194] Re-registered known tserver with Master: 20ea15da3bd74f2498765c4f9733de16 (127.15.185.67:33539)
I20260927 03:58:45.987324 23455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:45.972182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.124:46615 (local address 127.15.185.67:57477)
0927 03:58:45.972553 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:45.972600 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:45.973338 (+ 738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:45.973703 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:45.973708 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:45.974493 (+ 785us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:45.974580 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:45.974585 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.975624 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.975625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:45.978322 (+ 2697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:45.978324 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.979581 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.979583 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:45.979646 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:45.979960 (+ 314us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:45.979964 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:45.982015 (+ 2051us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:45.984280 (+ 2265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:45.984281 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:45.984282 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:45.985079 (+ 797us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:45.985295 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:45.985296 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:45.985296 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:45.985354 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:45.985667 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:45.985668 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:45.985708 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 03:58:45.986138 (+ 430us) client_negotiation.cc:240] Negotiation successful
0927 03:58:45.986255 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":1766,"thread_start_us":81,"threads_started":1}
I20260927 03:58:45.987401 22852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:45.972515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:39847 (local address 127.15.185.125:36857)
0927 03:58:45.972563 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:45.972566 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:45.973139 (+ 573us) server_negotiation.cc:407] Connection header received
0927 03:58:45.973584 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:45.973586 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:45.973597 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:45.973633 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:45.975947 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.976676 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.978439 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.978736 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.978783 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:45.982862 (+ 4079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:45.982864 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:45.982864 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:45.982872 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:45.985366 (+ 2494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:45.986573 (+ 1207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:45.986574 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:45.986574 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:45.986626 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:45.986810 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:45.986812 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:45.986812 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:45.986960 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:45.987033 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:45.987220 (+ 187us) server_negotiation.cc:299] Negotiation successful
0927 03:58:45.987296 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 03:58:45.987790 22203 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.185.67@KRBTEST.COM'} at 127.15.185.67:47583
I20260927 03:58:45.987968 23446 heartbeater.cc:347] Connected to a master server at 127.15.185.124:46615
I20260927 03:58:45.988018 23446 heartbeater.cc:464] Registering TS with master...
I20260927 03:58:45.988085 23446 heartbeater.cc:511] Master 127.15.185.124:46615 requested a full tablet report, sending...
I20260927 03:58:45.988358 22994 ts_manager.cc:194] Registered new tserver with Master: 20ea15da3bd74f2498765c4f9733de16 (127.15.185.67:33539)
I20260927 03:58:45.988471 23458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:45.972936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50642 (local address 127.15.185.125:36857)
0927 03:58:45.973412 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:45.973415 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:45.973799 (+ 384us) server_negotiation.cc:407] Connection header received
0927 03:58:45.973913 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:45.973914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:45.973925 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:45.973976 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:45.976230 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.976939 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.978486 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.978825 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.978870 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:45.984158 (+ 5288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:45.984159 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:45.984160 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:45.984166 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:45.986270 (+ 2104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:45.987121 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:45.987122 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:45.987123 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:45.987181 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:45.987557 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:45.987558 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:45.987559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:45.987687 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:45.987779 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:45.988273 (+ 494us) server_negotiation.cc:299] Negotiation successful
0927 03:58:45.988356 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":50,"threads_started":1}
I20260927 03:58:45.988739 23456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:45.972187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.125:36857 (local address 127.15.185.67:39847)
0927 03:58:45.973081 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:45.973091 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:45.973491 (+ 400us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:45.973704 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:45.973709 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:45.974330 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:45.975859 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:45.975861 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.976769 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.976770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:45.977545 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:45.977546 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.978862 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.978863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:45.978928 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:45.979807 (+ 879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:45.979811 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:45.982460 (+ 2649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:45.985499 (+ 3039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:45.985500 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:45.985500 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:45.986501 (+ 1001us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:45.986695 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:45.986696 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:45.986696 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:45.986753 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:45.987027 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:45.987028 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:45.987099 (+ 71us) client_negotiation.cc:769] Sending connection context
0927 03:58:45.988232 (+ 1133us) client_negotiation.cc:240] Negotiation successful
0927 03:58:45.988334 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":871,"mutex_wait_us":1677,"thread_start_us":64,"threads_started":1}
I20260927 03:58:45.989142 23448 heartbeater.cc:347] Connected to a master server at 127.15.185.125:36857
I20260927 03:58:45.989423 23448 heartbeater.cc:464] Registering TS with master...
I20260927 03:58:45.989487 23448 heartbeater.cc:511] Master 127.15.185.125:36857 requested a full tablet report, sending...
I20260927 03:58:45.989727 22801 ts_manager.cc:194] Registered new tserver with Master: 20ea15da3bd74f2498765c4f9733de16 (127.15.185.67:33539)
I20260927 03:58:45.989256 22203 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38348:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:58:45.991093 22203 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:58:45.998307 22761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:45.994622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.65:35459 (local address 127.0.0.1:51418)
0927 03:58:45.994679 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:45.994691 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:45.994732 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:45.995221 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:45.995222 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:45.995223 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:58:45.995270 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:45.995272 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.996280 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.996281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:45.997024 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:45.997026 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.998046 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.998048 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:45.998110 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:45.998114 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:58:45.998176 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 03:58:45.998216 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 03:58:45.998588 23469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:45.994831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51418 (local address 127.15.185.65:35459)
0927 03:58:45.995002 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:45.995007 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:45.995019 (+ 12us) server_negotiation.cc:407] Connection header received
0927 03:58:45.995086 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:45.995088 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:45.995105 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:45.995219 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:58:45.995371 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.996190 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.997139 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.997949 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.998031 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:45.998099 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:45.998287 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 03:58:45.998329 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":98,"threads_started":1}
I20260927 03:58:45.999317 23470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:45.994855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54246 (local address 127.15.185.67:33539)
0927 03:58:45.995333 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:45.995338 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:45.995345 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:58:45.995376 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:45.995378 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:45.995391 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:45.995491 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:58:45.995596 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.996692 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.998045 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.998926 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.999005 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:45.999098 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:45.999206 (+ 108us) server_negotiation.cc:299] Negotiation successful
0927 03:58:45.999243 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":78,"threads_started":1}
I20260927 03:58:45.999321 23467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:45.994652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.67:33539 (local address 127.0.0.1:54246)
0927 03:58:45.994790 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:45.994802 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:45.994851 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:45.995493 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:45.995493 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:45.995495 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:58:45.995534 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:45.995535 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.996831 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.996832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:45.997964 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:45.997966 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.999022 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.999023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:45.999126 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:45.999130 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:58:45.999192 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 03:58:45.999228 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":17,"thread_start_us":97,"threads_started":1}
I20260927 03:58:45.999351 23107 tablet_service.cc:1467] Processing CreateTablet for tablet 4cc9a2b0adf04dfabcf0ef06ab968441 (DEFAULT_TABLE table=test-table [id=65dfbe116bc8431394c6cc8e9edd8f27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:58:45.999788 23107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cc9a2b0adf04dfabcf0ef06ab968441. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:46.000046 23380 tablet_service.cc:1467] Processing CreateTablet for tablet 4cc9a2b0adf04dfabcf0ef06ab968441 (DEFAULT_TABLE table=test-table [id=65dfbe116bc8431394c6cc8e9edd8f27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:58:46.000357 23380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cc9a2b0adf04dfabcf0ef06ab968441. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:46.000679 23468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:45.994672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.66:40143 (local address 127.0.0.1:54006)
0927 03:58:45.995060 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:45.995070 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:45.995116 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:45.995449 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:45.995450 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:45.995452 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:58:45.995499 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:45.995500 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.997003 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.997005 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:45.999251 (+ 2246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:45.999254 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:46.000431 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:46.000432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:46.000499 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:46.000503 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:58:46.000567 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 03:58:46.000605 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":94,"threads_started":1}
I20260927 03:58:46.000725 23471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:45.994887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54006 (local address 127.15.185.66:40143)
0927 03:58:45.995223 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:45.995227 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:45.995236 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:58:45.995272 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:45.995274 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:45.995289 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:45.995360 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:58:45.995572 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:45.996918 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:45.999337 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:46.000328 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:46.000412 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:46.000485 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:46.000587 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 03:58:46.000630 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":73,"threads_started":1}
I20260927 03:58:46.002592 23472 tablet_bootstrap.cc:492] T 4cc9a2b0adf04dfabcf0ef06ab968441 P ce9010ea1dc94a0f91d6c8b10895c2f1: Bootstrap starting.
I20260927 03:58:46.003191 23239 tablet_service.cc:1467] Processing CreateTablet for tablet 4cc9a2b0adf04dfabcf0ef06ab968441 (DEFAULT_TABLE table=test-table [id=65dfbe116bc8431394c6cc8e9edd8f27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:58:46.003298 23472 tablet_bootstrap.cc:654] T 4cc9a2b0adf04dfabcf0ef06ab968441 P ce9010ea1dc94a0f91d6c8b10895c2f1: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:46.003490 23239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cc9a2b0adf04dfabcf0ef06ab968441. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:46.003603 23472 log.cc:824] T 4cc9a2b0adf04dfabcf0ef06ab968441 P ce9010ea1dc94a0f91d6c8b10895c2f1: Log is configured to *not* fsync() on all Append() calls
I20260927 03:58:46.003965 23473 tablet_bootstrap.cc:492] T 4cc9a2b0adf04dfabcf0ef06ab968441 P 20ea15da3bd74f2498765c4f9733de16: Bootstrap starting.
I20260927 03:58:46.004403 23472 tablet_bootstrap.cc:492] T 4cc9a2b0adf04dfabcf0ef06ab968441 P ce9010ea1dc94a0f91d6c8b10895c2f1: No bootstrap required, opened a new log
I20260927 03:58:46.004472 23472 ts_tablet_manager.cc:1397] T 4cc9a2b0adf04dfabcf0ef06ab968441 P ce9010ea1dc94a0f91d6c8b10895c2f1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:58:46.004710 23473 tablet_bootstrap.cc:654] T 4cc9a2b0adf04dfabcf0ef06ab968441 P 20ea15da3bd74f2498765c4f9733de16: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:46.004999 23473 log.cc:824] T 4cc9a2b0adf04dfabcf0ef06ab968441 P 20ea15da3bd74f2498765c4f9733de16: Log is configured to *not* fsync() on all Append() calls
I20260927 03:58:46.005787 23473 tablet_bootstrap.cc:492] T 4cc9a2b0adf04dfabcf0ef06ab968441 P 20ea15da3bd74f2498765c4f9733de16: No bootstrap required, opened a new log
I20260927 03:58:46.005863 23473 ts_tablet_manager.cc:1397] T 4cc9a2b0adf04dfabcf0ef06ab968441 P 20ea15da3bd74f2498765c4f9733de16: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:58:46.006853 23476 tablet_bootstrap.cc:492] T 4cc9a2b0adf04dfabcf0ef06ab968441 P 55d1a890d9f4473fa298231905647c4a: Bootstrap starting.
I20260927 03:58:46.006860 23472 raft_consensus.cc:348] T 4cc9a2b0adf04dfabcf0ef06ab968441 P ce9010ea1dc94a0f91d6c8b10895c2f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce9010ea1dc94a0f91d6c8b10895c2f1" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 35459 } } peers { permanent_uuid: "20ea15da3bd74f2498765c4f9733de16" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 33539 } } peers { permanent_uuid: "55d1a890d9f4473fa298231905647c4a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 40143 } }
I20260927 03:58:46.007110 23472 raft_consensus.cc:374] T 4cc9a2b0adf04dfabcf0ef06ab968441 P ce9010ea1dc94a0f91d6c8b10895c2f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:46.007164 23472 raft_consensus.cc:729] T 4cc9a2b0adf04dfabcf0ef06ab968441 P ce9010ea1dc94a0f91d6c8b10895c2f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce9010ea1dc94a0f91d6c8b10895c2f1, State: Initialized, Role: FOLLOWER
I20260927 03:58:46.007275 23472 consensus_queue.cc:260] T 4cc9a2b0adf04dfabcf0ef06ab968441 P ce9010ea1dc94a0f91d6c8b10895c2f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce9010ea1dc94a0f91d6c8b10895c2f1" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 35459 } } peers { permanent_uuid: "20ea15da3bd74f2498765c4f9733de16" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 33539 } } peers { permanent_uuid: "55d1a890d9f4473fa298231905647c4a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 40143 } }
I20260927 03:58:46.007524 23472 ts_tablet_manager.cc:1428] T 4cc9a2b0adf04dfabcf0ef06ab968441 P ce9010ea1dc94a0f91d6c8b10895c2f1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:58:46.007660 23174 heartbeater.cc:503] Master 127.15.185.126:43909 was elected leader, sending a full tablet report...
I20260927 03:58:46.007772 23476 tablet_bootstrap.cc:654] T 4cc9a2b0adf04dfabcf0ef06ab968441 P 55d1a890d9f4473fa298231905647c4a: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:46.008124 23476 log.cc:824] T 4cc9a2b0adf04dfabcf0ef06ab968441 P 55d1a890d9f4473fa298231905647c4a: Log is configured to *not* fsync() on all Append() calls
I20260927 03:58:46.009019 23476 tablet_bootstrap.cc:492] T 4cc9a2b0adf04dfabcf0ef06ab968441 P 55d1a890d9f4473fa298231905647c4a: No bootstrap required, opened a new log
I20260927 03:58:46.009120 23476 ts_tablet_manager.cc:1397] T 4cc9a2b0adf04dfabcf0ef06ab968441 P 55d1a890d9f4473fa298231905647c4a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:58:46.009035 23473 raft_consensus.cc:348] T 4cc9a2b0adf04dfabcf0ef06ab968441 P 20ea15da3bd74f2498765c4f9733de16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce9010ea1dc94a0f91d6c8b10895c2f1" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 35459 } } peers { permanent_uuid: "20ea15da3bd74f2498765c4f9733de16" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 33539 } } peers { permanent_uuid: "55d1a890d9f4473fa298231905647c4a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 40143 } }
I20260927 03:58:46.009207 23473 raft_consensus.cc:374] T 4cc9a2b0adf04dfabcf0ef06ab968441 P 20ea15da3bd74f2498765c4f9733de16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:46.009253 23473 raft_consensus.cc:729] T 4cc9a2b0adf04dfabcf0ef06ab968441 P 20ea15da3bd74f2498765c4f9733de16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20ea15da3bd74f2498765c4f9733de16, State: Initialized, Role: FOLLOWER
I20260927 03:58:46.009368 23473 consensus_queue.cc:260] T 4cc9a2b0adf04dfabcf0ef06ab968441 P 20ea15da3bd74f2498765c4f9733de16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce9010ea1dc94a0f91d6c8b10895c2f1" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 35459 } } peers { permanent_uuid: "20ea15da3bd74f2498765c4f9733de16" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 33539 } } peers { permanent_uuid: "55d1a890d9f4473fa298231905647c4a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 40143 } }
I20260927 03:58:46.009670 23447 heartbeater.cc:503] Master 127.15.185.126:43909 was elected leader, sending a full tablet report...
I20260927 03:58:46.009667 23473 ts_tablet_manager.cc:1428] T 4cc9a2b0adf04dfabcf0ef06ab968441 P 20ea15da3bd74f2498765c4f9733de16: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260927 03:58:46.011492 23476 raft_consensus.cc:348] T 4cc9a2b0adf04dfabcf0ef06ab968441 P 55d1a890d9f4473fa298231905647c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce9010ea1dc94a0f91d6c8b10895c2f1" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 35459 } } peers { permanent_uuid: "20ea15da3bd74f2498765c4f9733de16" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 33539 } } peers { permanent_uuid: "55d1a890d9f4473fa298231905647c4a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 40143 } }
I20260927 03:58:46.011644 23476 raft_consensus.cc:374] T 4cc9a2b0adf04dfabcf0ef06ab968441 P 55d1a890d9f4473fa298231905647c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:46.011713 23476 raft_consensus.cc:729] T 4cc9a2b0adf04dfabcf0ef06ab968441 P 55d1a890d9f4473fa298231905647c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55d1a890d9f4473fa298231905647c4a, State: Initialized, Role: FOLLOWER
I20260927 03:58:46.011827 23476 consensus_queue.cc:260] T 4cc9a2b0adf04dfabcf0ef06ab968441 P 55d1a890d9f4473fa298231905647c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce9010ea1dc94a0f91d6c8b10895c2f1" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 35459 } } peers { permanent_uuid: "20ea15da3bd74f2498765c4f9733de16" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 33539 } } peers { permanent_uuid: "55d1a890d9f4473fa298231905647c4a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 40143 } }
I20260927 03:58:46.012068 23476 ts_tablet_manager.cc:1428] T 4cc9a2b0adf04dfabcf0ef06ab968441 P 55d1a890d9f4473fa298231905647c4a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:58:46.012135 23311 heartbeater.cc:503] Master 127.15.185.126:43909 was elected leader, sending a full tablet report...
I20260927 03:58:46.020988 23468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:46.018587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.124:46615 (local address 127.0.0.1:47650)
0927 03:58:46.018697 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:46.018709 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:46.018796 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:46.018943 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:46.018944 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:46.018945 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:58:46.018989 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:46.018990 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:46.019630 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:46.019630 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:46.020328 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:46.020330 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:46.020816 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:46.020816 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:46.020856 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:46.020860 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:58:46.020902 (+ 42us) client_negotiation.cc:240] Negotiation successful
0927 03:58:46.020935 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 03:58:46.021006 23178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:46.018627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47650 (local address 127.15.185.124:46615)
0927 03:58:46.018669 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:46.018671 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:46.018774 (+ 103us) server_negotiation.cc:407] Connection header received
0927 03:58:46.018860 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:46.018860 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:46.018879 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:46.018932 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:58:46.019087 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:46.019567 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:46.020396 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:46.020754 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:46.020810 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:46.020857 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:46.020908 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 03:58:46.020930 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 03:58:46.021306 23003 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260927 03:58:46.021659 23033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1868152744144c8f97d2a16fc4d34438. Latest consensus state: current_term: 1 leader_uuid: "1868152744144c8f97d2a16fc4d34438" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1868152744144c8f97d2a16fc4d34438" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 43909 } } peers { permanent_uuid: "87685032f24e44c59ad7c48aaff26803" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 36857 } attrs { promote: false } } peers { permanent_uuid: "08795fc9741d45d3953bb7183c4cb80f" member_type: NON_VOTER last_known_addr { host: "127.15.185.124" port: 46615 } attrs { promote: true } } }
I20260927 03:58:46.021759 23033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f [sys.catalog]: This master's current role is: LEARNER
W20260927 03:58:46.081140 23313 tablet.cc:2369] T 4cc9a2b0adf04dfabcf0ef06ab968441 P 55d1a890d9f4473fa298231905647c4a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 03:58:46.189332 23176 tablet.cc:2369] T 4cc9a2b0adf04dfabcf0ef06ab968441 P ce9010ea1dc94a0f91d6c8b10895c2f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 03:58:46.222946 23449 tablet.cc:2369] T 4cc9a2b0adf04dfabcf0ef06ab968441 P 20ea15da3bd74f2498765c4f9733de16: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:58:46.374603 23477 raft_consensus.cc:482] T 4cc9a2b0adf04dfabcf0ef06ab968441 P ce9010ea1dc94a0f91d6c8b10895c2f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:58:46.374727 23477 raft_consensus.cc:504] T 4cc9a2b0adf04dfabcf0ef06ab968441 P ce9010ea1dc94a0f91d6c8b10895c2f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce9010ea1dc94a0f91d6c8b10895c2f1" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 35459 } } peers { permanent_uuid: "20ea15da3bd74f2498765c4f9733de16" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 33539 } } peers { permanent_uuid: "55d1a890d9f4473fa298231905647c4a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 40143 } }
I20260927 03:58:46.375176 23477 leader_election.cc:288] T 4cc9a2b0adf04dfabcf0ef06ab968441 P ce9010ea1dc94a0f91d6c8b10895c2f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20ea15da3bd74f2498765c4f9733de16 (127.15.185.67:33539), 55d1a890d9f4473fa298231905647c4a (127.15.185.66:40143)
I20260927 03:58:46.378068 23471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:46.375506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:43693 (local address 127.15.185.66:40143)
0927 03:58:46.375566 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:46.375568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:46.375620 (+ 52us) server_negotiation.cc:407] Connection header received
0927 03:58:46.375695 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:46.375695 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:46.375722 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:46.375774 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:58:46.375945 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:46.376384 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:46.377353 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:46.377716 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:46.377771 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:46.377778 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:46.377958 (+ 180us) server_negotiation.cc:299] Negotiation successful
0927 03:58:46.378002 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 03:58:46.378079 23485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:46.375378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.66:40143 (local address 127.15.185.65:43693)
0927 03:58:46.375544 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:46.375556 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:46.375637 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:46.375796 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:46.375797 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:46.375800 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:58:46.375851 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:46.375869 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:46.376470 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:46.376471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:46.377257 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:46.377259 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:46.377802 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:46.377803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:46.377878 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:46.377883 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:58:46.377930 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 03:58:46.377965 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":3,"thread_start_us":62,"threads_started":1}
I20260927 03:58:46.378171 23484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:46.375378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.67:33539 (local address 127.15.185.65:44569)
0927 03:58:46.375506 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:46.375519 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:46.375578 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:46.375744 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:46.375745 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:46.375747 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:58:46.375822 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:46.375824 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:46.376473 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:46.376474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:46.377262 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:46.377264 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:46.377855 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:46.377861 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:46.377935 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:46.377939 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:58:46.377998 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 03:58:46.378040 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":40,"thread_start_us":66,"threads_started":1}
I20260927 03:58:46.378342 23470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:46.375491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:44569 (local address 127.15.185.67:33539)
0927 03:58:46.375550 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:46.375552 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:46.375566 (+ 14us) server_negotiation.cc:407] Connection header received
0927 03:58:46.375657 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:46.375658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:46.375668 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:46.375726 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:58:46.375898 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:46.376384 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:46.377387 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:46.377736 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:46.377831 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:46.377839 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:46.378021 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 03:58:46.378057 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 03:58:46.378422 23249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cc9a2b0adf04dfabcf0ef06ab968441" candidate_uuid: "ce9010ea1dc94a0f91d6c8b10895c2f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55d1a890d9f4473fa298231905647c4a" is_pre_election: true
I20260927 03:58:46.378599 23249 raft_consensus.cc:2454] T 4cc9a2b0adf04dfabcf0ef06ab968441 P 55d1a890d9f4473fa298231905647c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce9010ea1dc94a0f91d6c8b10895c2f1 in term 0.
I20260927 03:58:46.378600 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cc9a2b0adf04dfabcf0ef06ab968441" candidate_uuid: "ce9010ea1dc94a0f91d6c8b10895c2f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20ea15da3bd74f2498765c4f9733de16" is_pre_election: true
I20260927 03:58:46.378765 23400 raft_consensus.cc:2454] T 4cc9a2b0adf04dfabcf0ef06ab968441 P 20ea15da3bd74f2498765c4f9733de16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce9010ea1dc94a0f91d6c8b10895c2f1 in term 0.
I20260927 03:58:46.378850 23062 leader_election.cc:302] T 4cc9a2b0adf04dfabcf0ef06ab968441 P ce9010ea1dc94a0f91d6c8b10895c2f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55d1a890d9f4473fa298231905647c4a, ce9010ea1dc94a0f91d6c8b10895c2f1; no voters:
I20260927 03:58:46.378996 23477 raft_consensus.cc:2781] T 4cc9a2b0adf04dfabcf0ef06ab968441 P ce9010ea1dc94a0f91d6c8b10895c2f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:58:46.379086 23477 raft_consensus.cc:482] T 4cc9a2b0adf04dfabcf0ef06ab968441 P ce9010ea1dc94a0f91d6c8b10895c2f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:58:46.379117 23477 raft_consensus.cc:3037] T 4cc9a2b0adf04dfabcf0ef06ab968441 P ce9010ea1dc94a0f91d6c8b10895c2f1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:46.379801 23477 raft_consensus.cc:504] T 4cc9a2b0adf04dfabcf0ef06ab968441 P ce9010ea1dc94a0f91d6c8b10895c2f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce9010ea1dc94a0f91d6c8b10895c2f1" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 35459 } } peers { permanent_uuid: "20ea15da3bd74f2498765c4f9733de16" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 33539 } } peers { permanent_uuid: "55d1a890d9f4473fa298231905647c4a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 40143 } }
I20260927 03:58:46.379943 23477 leader_election.cc:288] T 4cc9a2b0adf04dfabcf0ef06ab968441 P ce9010ea1dc94a0f91d6c8b10895c2f1 [CANDIDATE]: Term 1 election: Requested vote from peers 20ea15da3bd74f2498765c4f9733de16 (127.15.185.67:33539), 55d1a890d9f4473fa298231905647c4a (127.15.185.66:40143)
I20260927 03:58:46.380102 23249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cc9a2b0adf04dfabcf0ef06ab968441" candidate_uuid: "ce9010ea1dc94a0f91d6c8b10895c2f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55d1a890d9f4473fa298231905647c4a"
I20260927 03:58:46.380122 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cc9a2b0adf04dfabcf0ef06ab968441" candidate_uuid: "ce9010ea1dc94a0f91d6c8b10895c2f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20ea15da3bd74f2498765c4f9733de16"
I20260927 03:58:46.380178 23249 raft_consensus.cc:3037] T 4cc9a2b0adf04dfabcf0ef06ab968441 P 55d1a890d9f4473fa298231905647c4a [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:46.380184 23400 raft_consensus.cc:3037] T 4cc9a2b0adf04dfabcf0ef06ab968441 P 20ea15da3bd74f2498765c4f9733de16 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:46.381014 23400 raft_consensus.cc:2454] T 4cc9a2b0adf04dfabcf0ef06ab968441 P 20ea15da3bd74f2498765c4f9733de16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce9010ea1dc94a0f91d6c8b10895c2f1 in term 1.
I20260927 03:58:46.381014 23249 raft_consensus.cc:2454] T 4cc9a2b0adf04dfabcf0ef06ab968441 P 55d1a890d9f4473fa298231905647c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce9010ea1dc94a0f91d6c8b10895c2f1 in term 1.
I20260927 03:58:46.381172 23063 leader_election.cc:302] T 4cc9a2b0adf04dfabcf0ef06ab968441 P ce9010ea1dc94a0f91d6c8b10895c2f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20ea15da3bd74f2498765c4f9733de16, ce9010ea1dc94a0f91d6c8b10895c2f1; no voters:
I20260927 03:58:46.381278 23477 raft_consensus.cc:2781] T 4cc9a2b0adf04dfabcf0ef06ab968441 P ce9010ea1dc94a0f91d6c8b10895c2f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:58:46.381394 23477 raft_consensus.cc:686] T 4cc9a2b0adf04dfabcf0ef06ab968441 P ce9010ea1dc94a0f91d6c8b10895c2f1 [term 1 LEADER]: Becoming Leader. State: Replica: ce9010ea1dc94a0f91d6c8b10895c2f1, State: Running, Role: LEADER
I20260927 03:58:46.381518 23477 consensus_queue.cc:237] T 4cc9a2b0adf04dfabcf0ef06ab968441 P ce9010ea1dc94a0f91d6c8b10895c2f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce9010ea1dc94a0f91d6c8b10895c2f1" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 35459 } } peers { permanent_uuid: "20ea15da3bd74f2498765c4f9733de16" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 33539 } } peers { permanent_uuid: "55d1a890d9f4473fa298231905647c4a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 40143 } }
I20260927 03:58:46.382270 22203 catalog_manager.cc:5495] T 4cc9a2b0adf04dfabcf0ef06ab968441 P ce9010ea1dc94a0f91d6c8b10895c2f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce9010ea1dc94a0f91d6c8b10895c2f1 (127.15.185.65). New cstate: current_term: 1 leader_uuid: "ce9010ea1dc94a0f91d6c8b10895c2f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce9010ea1dc94a0f91d6c8b10895c2f1" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20ea15da3bd74f2498765c4f9733de16" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55d1a890d9f4473fa298231905647c4a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 40143 } health_report { overall_health: UNKNOWN } } }
I20260927 03:58:46.382928 22763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1868152744144c8f97d2a16fc4d34438 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1868152744144c8f97d2a16fc4d34438" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1868152744144c8f97d2a16fc4d34438" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 43909 } } peers { permanent_uuid: "87685032f24e44c59ad7c48aaff26803" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 36857 } attrs { promote: false } } peers { permanent_uuid: "08795fc9741d45d3953bb7183c4cb80f" member_type: NON_VOTER last_known_addr { host: "127.15.185.124" port: 46615 } attrs { promote: true } } }
I20260927 03:58:46.383103 22763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1868152744144c8f97d2a16fc4d34438 [sys.catalog]: This master's current role is: LEADER
I20260927 03:58:46.383735 22763 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 1868152744144c8f97d2a16fc4d34438: attempting to promote NON_VOTER 08795fc9741d45d3953bb7183c4cb80f to VOTER
I20260927 03:58:46.383846 22763 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1868152744144c8f97d2a16fc4d34438 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1868152744144c8f97d2a16fc4d34438" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 43909 } } peers { permanent_uuid: "87685032f24e44c59ad7c48aaff26803" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 36857 } attrs { promote: false } } peers { permanent_uuid: "08795fc9741d45d3953bb7183c4cb80f" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 46615 } attrs { promote: false } }
I20260927 03:58:46.384195 22811 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803 [term 1 FOLLOWER]: Refusing update from remote peer 1868152744144c8f97d2a16fc4d34438: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260927 03:58:46.384328 23003 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f [term 1 LEARNER]: Refusing update from remote peer 1868152744144c8f97d2a16fc4d34438: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260927 03:58:46.384356 23489 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1868152744144c8f97d2a16fc4d34438 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87685032f24e44c59ad7c48aaff26803" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 36857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260927 03:58:46.384476 23489 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1868152744144c8f97d2a16fc4d34438 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08795fc9741d45d3953bb7183c4cb80f" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 46615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260927 03:58:46.384963 22763 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1868152744144c8f97d2a16fc4d34438 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 08795fc9741d45d3953bb7183c4cb80f (127.15.185.124) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1868152744144c8f97d2a16fc4d34438" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 43909 } } peers { permanent_uuid: "87685032f24e44c59ad7c48aaff26803" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 36857 } attrs { promote: false } } peers { permanent_uuid: "08795fc9741d45d3953bb7183c4cb80f" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 46615 } attrs { promote: false } } }
I20260927 03:58:46.385113 23003 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 08795fc9741d45d3953bb7183c4cb80f (127.15.185.124) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1868152744144c8f97d2a16fc4d34438" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 43909 } } peers { permanent_uuid: "87685032f24e44c59ad7c48aaff26803" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 36857 } attrs { promote: false } } peers { permanent_uuid: "08795fc9741d45d3953bb7183c4cb80f" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 46615 } attrs { promote: false } } }
I20260927 03:58:46.385789 22763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1868152744144c8f97d2a16fc4d34438 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1868152744144c8f97d2a16fc4d34438" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1868152744144c8f97d2a16fc4d34438" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 43909 } } peers { permanent_uuid: "87685032f24e44c59ad7c48aaff26803" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 36857 } attrs { promote: false } } peers { permanent_uuid: "08795fc9741d45d3953bb7183c4cb80f" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 46615 } attrs { promote: false } } }
I20260927 03:58:46.385820 23489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1868152744144c8f97d2a16fc4d34438 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1868152744144c8f97d2a16fc4d34438" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1868152744144c8f97d2a16fc4d34438" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 43909 } } peers { permanent_uuid: "87685032f24e44c59ad7c48aaff26803" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 36857 } attrs { promote: false } } peers { permanent_uuid: "08795fc9741d45d3953bb7183c4cb80f" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 46615 } attrs { promote: false } } }
I20260927 03:58:46.385986 23033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1868152744144c8f97d2a16fc4d34438" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1868152744144c8f97d2a16fc4d34438" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 43909 } } peers { permanent_uuid: "87685032f24e44c59ad7c48aaff26803" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 36857 } attrs { promote: false } } peers { permanent_uuid: "08795fc9741d45d3953bb7183c4cb80f" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 46615 } attrs { promote: false } } }
I20260927 03:58:46.385854 23488 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1868152744144c8f97d2a16fc4d34438 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1868152744144c8f97d2a16fc4d34438" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1868152744144c8f97d2a16fc4d34438" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 43909 } } peers { permanent_uuid: "87685032f24e44c59ad7c48aaff26803" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 36857 } attrs { promote: false } } peers { permanent_uuid: "08795fc9741d45d3953bb7183c4cb80f" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 46615 } attrs { promote: false } } }
I20260927 03:58:46.386059 23033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:58:46.386073 23488 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1868152744144c8f97d2a16fc4d34438 [sys.catalog]: This master's current role is: LEADER
I20260927 03:58:46.386018 22811 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 08795fc9741d45d3953bb7183c4cb80f (127.15.185.124) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1868152744144c8f97d2a16fc4d34438" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 43909 } } peers { permanent_uuid: "87685032f24e44c59ad7c48aaff26803" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 36857 } attrs { promote: false } } peers { permanent_uuid: "08795fc9741d45d3953bb7183c4cb80f" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 46615 } attrs { promote: false } } }
I20260927 03:58:46.385880 22763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1868152744144c8f97d2a16fc4d34438 [sys.catalog]: This master's current role is: LEADER
I20260927 03:58:46.386919 23491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1868152744144c8f97d2a16fc4d34438" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1868152744144c8f97d2a16fc4d34438" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 43909 } } peers { permanent_uuid: "87685032f24e44c59ad7c48aaff26803" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 36857 } attrs { promote: false } } peers { permanent_uuid: "08795fc9741d45d3953bb7183c4cb80f" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 46615 } attrs { promote: false } } }
I20260927 03:58:46.385888 23489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1868152744144c8f97d2a16fc4d34438 [sys.catalog]: This master's current role is: LEADER
I20260927 03:58:46.387108 23491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:58:46.441547 23469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:46.438863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51432 (local address 127.15.185.65:35459)
0927 03:58:46.438933 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:46.438935 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:46.438942 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:58:46.439011 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:46.439011 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:46.439029 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:46.439127 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:58:46.439287 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:46.439841 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:46.440720 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:46.440931 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:46.441011 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:46.441198 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:58:46.441309 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:58:46.441403 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:46.441421 (+ 18us) server_negotiation.cc:299] Negotiation successful
0927 03:58:46.441441 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 03:58:46.443367 23249 raft_consensus.cc:1261] T 4cc9a2b0adf04dfabcf0ef06ab968441 P 55d1a890d9f4473fa298231905647c4a [term 1 FOLLOWER]: Refusing update from remote peer ce9010ea1dc94a0f91d6c8b10895c2f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:58:46.443367 23400 raft_consensus.cc:1261] T 4cc9a2b0adf04dfabcf0ef06ab968441 P 20ea15da3bd74f2498765c4f9733de16 [term 1 FOLLOWER]: Refusing update from remote peer ce9010ea1dc94a0f91d6c8b10895c2f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:58:46.443727 23477 consensus_queue.cc:1035] T 4cc9a2b0adf04dfabcf0ef06ab968441 P ce9010ea1dc94a0f91d6c8b10895c2f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55d1a890d9f4473fa298231905647c4a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 40143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:58:46.443851 23486 consensus_queue.cc:1035] T 4cc9a2b0adf04dfabcf0ef06ab968441 P ce9010ea1dc94a0f91d6c8b10895c2f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20ea15da3bd74f2498765c4f9733de16" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:58:46.445969 23493 mvcc.cc:205] Tried to move back new op lower bound from 7333812332309225472 to 7333812332059262976. Current Snapshot: MvccSnapshot[applied={T|T < 7333812332309225472}]
I20260927 03:58:46.446660 23495 mvcc.cc:205] Tried to move back new op lower bound from 7333812332309225472 to 7333812332059262976. Current Snapshot: MvccSnapshot[applied={T|T < 7333812332309225472}]
I20260927 03:58:46.447867 23494 mvcc.cc:205] Tried to move back new op lower bound from 7333812332309225472 to 7333812332059262976. Current Snapshot: MvccSnapshot[applied={T|T < 7333812332309225472}]
I20260927 03:58:46.449910 22203 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:58:46.449996 22203 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:58:46.451371 22203 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1868152744144c8f97d2a16fc4d34438: Sending DeleteTablet for 3 replicas of tablet 4cc9a2b0adf04dfabcf0ef06ab968441
I20260927 03:58:46.451719 23380 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cc9a2b0adf04dfabcf0ef06ab968441 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:58:46 UTC) from {username='oryx', principal='oryx/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:54246
I20260927 03:58:46.451740 23107 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cc9a2b0adf04dfabcf0ef06ab968441 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:58:46 UTC) from {username='oryx', principal='oryx/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:51418
I20260927 03:58:46.451738 23239 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cc9a2b0adf04dfabcf0ef06ab968441 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:58:46 UTC) from {username='oryx', principal='oryx/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:54006
I20260927 03:58:46.452132 23502 tablet_replica.cc:330] T 4cc9a2b0adf04dfabcf0ef06ab968441 P ce9010ea1dc94a0f91d6c8b10895c2f1: stopping tablet replica
I20260927 03:58:46.452227 23502 raft_consensus.cc:2229] T 4cc9a2b0adf04dfabcf0ef06ab968441 P ce9010ea1dc94a0f91d6c8b10895c2f1 [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:58:46.452353 23502 raft_consensus.cc:2258] T 4cc9a2b0adf04dfabcf0ef06ab968441 P ce9010ea1dc94a0f91d6c8b10895c2f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:58:46.452600 23503 tablet_replica.cc:330] T 4cc9a2b0adf04dfabcf0ef06ab968441 P 20ea15da3bd74f2498765c4f9733de16: stopping tablet replica
I20260927 03:58:46.452694 23503 raft_consensus.cc:2229] T 4cc9a2b0adf04dfabcf0ef06ab968441 P 20ea15da3bd74f2498765c4f9733de16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:58:46.452790 23503 raft_consensus.cc:2258] T 4cc9a2b0adf04dfabcf0ef06ab968441 P 20ea15da3bd74f2498765c4f9733de16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:58:46.453105 23502 ts_tablet_manager.cc:1905] T 4cc9a2b0adf04dfabcf0ef06ab968441 P ce9010ea1dc94a0f91d6c8b10895c2f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:58:46.453120 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 22171
I20260927 03:58:46.455165 23502 ts_tablet_manager.cc:1918] T 4cc9a2b0adf04dfabcf0ef06ab968441 P ce9010ea1dc94a0f91d6c8b10895c2f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:58:46.455240 23502 log.cc:1196] T 4cc9a2b0adf04dfabcf0ef06ab968441 P ce9010ea1dc94a0f91d6c8b10895c2f1: Deleting WAL directory at /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-0/wal/wals/4cc9a2b0adf04dfabcf0ef06ab968441
I20260927 03:58:46.455510 23502 ts_tablet_manager.cc:1939] T 4cc9a2b0adf04dfabcf0ef06ab968441 P ce9010ea1dc94a0f91d6c8b10895c2f1: Deleting consensus metadata
W20260927 03:58:46.460376 23063 connection.cc:535] server connection from 127.0.0.1:51418 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 03:58:46.452069 23504 tablet_replica.cc:330] T 4cc9a2b0adf04dfabcf0ef06ab968441 P 55d1a890d9f4473fa298231905647c4a: stopping tablet replica
I20260927 03:58:46.461150 23504 raft_consensus.cc:2229] T 4cc9a2b0adf04dfabcf0ef06ab968441 P 55d1a890d9f4473fa298231905647c4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:58:46.461270 23504 raft_consensus.cc:2258] T 4cc9a2b0adf04dfabcf0ef06ab968441 P 55d1a890d9f4473fa298231905647c4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:58:46.461445 23178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:46.452755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47662 (local address 127.15.185.124:46615)
0927 03:58:46.452796 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:46.452798 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:46.452805 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:58:46.452824 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:46.452825 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:46.452835 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:46.452884 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:46.454277 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:46.455017 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:46.456262 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:46.456631 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:46.456702 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:46.458253 (+ 1551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:46.458255 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:46.458256 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:46.458263 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:46.460154 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:46.460555 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:46.460556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:46.460557 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:46.460612 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:46.460819 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:46.460820 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:46.460820 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:46.460964 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:46.461041 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:46.461256 (+ 215us) server_negotiation.cc:299] Negotiation successful
0927 03:58:46.461331 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 03:58:46.461815 23503 ts_tablet_manager.cc:1905] T 4cc9a2b0adf04dfabcf0ef06ab968441 P 20ea15da3bd74f2498765c4f9733de16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:58:46.463377 23503 ts_tablet_manager.cc:1918] T 4cc9a2b0adf04dfabcf0ef06ab968441 P 20ea15da3bd74f2498765c4f9733de16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:58:46.463498 23503 log.cc:1196] T 4cc9a2b0adf04dfabcf0ef06ab968441 P 20ea15da3bd74f2498765c4f9733de16: Deleting WAL directory at /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-2/wal/wals/4cc9a2b0adf04dfabcf0ef06ab968441
I20260927 03:58:46.463479 23504 ts_tablet_manager.cc:1905] T 4cc9a2b0adf04dfabcf0ef06ab968441 P 55d1a890d9f4473fa298231905647c4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:58:46.463819 23503 ts_tablet_manager.cc:1939] T 4cc9a2b0adf04dfabcf0ef06ab968441 P 20ea15da3bd74f2498765c4f9733de16: Deleting consensus metadata
W20260927 03:58:46.464193 23334 connection.cc:441] server connection from 127.0.0.1:54246 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54246 (request call id 1) could send its response
I20260927 03:58:46.465281 23504 ts_tablet_manager.cc:1918] T 4cc9a2b0adf04dfabcf0ef06ab968441 P 55d1a890d9f4473fa298231905647c4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:58:46.465360 23504 log.cc:1196] T 4cc9a2b0adf04dfabcf0ef06ab968441 P 55d1a890d9f4473fa298231905647c4a: Deleting WAL directory at /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-1/wal/wals/4cc9a2b0adf04dfabcf0ef06ab968441
I20260927 03:58:46.465654 23504 ts_tablet_manager.cc:1939] T 4cc9a2b0adf04dfabcf0ef06ab968441 P 55d1a890d9f4473fa298231905647c4a: Deleting consensus metadata
W20260927 03:58:46.465988 23197 connection.cc:441] server connection from 127.0.0.1:54006 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54006 (request call id 1) could send its response
I20260927 03:58:46.472059 23178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:46.463585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47664 (local address 127.15.185.124:46615)
0927 03:58:46.463620 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:46.463622 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:46.464218 (+ 596us) server_negotiation.cc:407] Connection header received
0927 03:58:46.464311 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:46.464312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:46.464322 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:46.464350 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:46.465608 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:46.466278 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:46.467392 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:46.467601 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:46.467689 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:46.469182 (+ 1493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:46.469183 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:46.469184 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:46.469190 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:46.470712 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:46.471151 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:46.471152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:46.471153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:46.471213 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:46.471476 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:46.471478 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:46.471478 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:46.471646 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:46.471750 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:46.471870 (+ 120us) server_negotiation.cc:299] Negotiation successful
0927 03:58:46.471943 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 03:58:46.473358 23458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:46.463902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50656 (local address 127.15.185.125:36857)
0927 03:58:46.463938 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:46.463940 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:46.465538 (+ 1598us) server_negotiation.cc:407] Connection header received
0927 03:58:46.465571 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:46.465572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:46.465586 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:46.465645 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:46.466448 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:46.467204 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:46.468248 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:46.468572 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:46.468654 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:46.469921 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:46.469922 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:46.469923 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:46.469929 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:46.472088 (+ 2159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:46.472571 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:46.472572 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:46.472572 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:46.472625 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:46.472861 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:46.472862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:46.472862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:46.472974 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:46.473076 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:46.473182 (+ 106us) server_negotiation.cc:299] Negotiation successful
0927 03:58:46.473260 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 03:58:47.447593 23513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:47.447298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:43909 (local address 127.15.185.66:53573)
0927 03:58:47.447434 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:47.447531 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:43909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":60,"threads_started":1}
W20260927 03:58:47.447862 23311 heartbeater.cc:644] Failed to heartbeat to 127.15.185.126:43909 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.185.126:43909: connect: Connection refused (error 111)
I20260927 03:58:47.448735 23514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:47.448515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:43909 (local address 127.15.185.67:45107)
0927 03:58:47.448623 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:47.448692 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:43909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":55,"threads_started":1}
I20260927 03:58:47.448808 23515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:47.448517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:43909 (local address 127.15.185.65:48613)
0927 03:58:47.448701 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:47.448763 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:43909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":61,"threads_started":1}
W20260927 03:58:47.448984 23174 heartbeater.cc:644] Failed to heartbeat to 127.15.185.126:43909 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.185.126:43909: connect: Connection refused (error 111)
W20260927 03:58:47.448984 23447 heartbeater.cc:644] Failed to heartbeat to 127.15.185.126:43909 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.185.126:43909: connect: Connection refused (error 111)
I20260927 03:58:47.951745 23516 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1868152744144c8f97d2a16fc4d34438)
I20260927 03:58:47.951853 23517 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1868152744144c8f97d2a16fc4d34438)
I20260927 03:58:47.951870 23516 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1868152744144c8f97d2a16fc4d34438" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 43909 } } peers { permanent_uuid: "87685032f24e44c59ad7c48aaff26803" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 36857 } attrs { promote: false } } peers { permanent_uuid: "08795fc9741d45d3953bb7183c4cb80f" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 46615 } attrs { promote: false } }
I20260927 03:58:47.952001 23517 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1868152744144c8f97d2a16fc4d34438" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 43909 } } peers { permanent_uuid: "87685032f24e44c59ad7c48aaff26803" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 36857 } attrs { promote: false } } peers { permanent_uuid: "08795fc9741d45d3953bb7183c4cb80f" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 46615 } attrs { promote: false } }
I20260927 03:58:47.952442 23517 leader_election.cc:288] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1868152744144c8f97d2a16fc4d34438 (127.15.185.126:43909), 87685032f24e44c59ad7c48aaff26803 (127.15.185.125:36857)
I20260927 03:58:47.952637 23518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:47.952356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:43909 (local address 127.0.0.1:38846)
0927 03:58:47.952480 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:47.952587 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:43909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":43,"threads_started":1}
I20260927 03:58:47.952642 23516 leader_election.cc:288] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1868152744144c8f97d2a16fc4d34438 (127.15.185.126:43909), 08795fc9741d45d3953bb7183c4cb80f (127.15.185.124:46615)
I20260927 03:58:47.952955 23520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:47.952369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:43909 (local address 127.0.0.1:38842)
0927 03:58:47.952868 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:47.952920 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:43909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":44,"threads_started":1}
W20260927 03:58:47.953099 22786 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.185.126:43909: connect: Connection refused (error 111)
W20260927 03:58:47.953334 22979 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.185.126:43909: connect: Connection refused (error 111)
I20260927 03:58:47.953522 23520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:47.953439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:43909 (local address 127.0.0.1:38850)
0927 03:58:47.953476 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:47.953500 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:43909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260927 03:58:47.953598 22786 leader_election.cc:334] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1868152744144c8f97d2a16fc4d34438 (127.15.185.126:43909): Network error: Client connection negotiation failed: client connection to 127.15.185.126:43909: connect: Connection refused (error 111)
I20260927 03:58:47.953949 23518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:47.953853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:43909 (local address 127.0.0.1:38864)
0927 03:58:47.953900 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:47.953930 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:43909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 03:58:47.954020 22979 leader_election.cc:334] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1868152744144c8f97d2a16fc4d34438 (127.15.185.126:43909): Network error: Client connection negotiation failed: client connection to 127.15.185.126:43909: connect: Connection refused (error 111)
I20260927 03:58:47.956357 23522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:47.952563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47666 (local address 127.15.185.124:46615)
0927 03:58:47.952850 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:47.952852 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:47.952861 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:58:47.952887 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:47.952887 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:47.952899 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:47.952976 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:58:47.953187 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:47.953976 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:47.955400 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:47.955967 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:47.956052 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:47.956062 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:47.956222 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 03:58:47.956264 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":57,"threads_started":1}
I20260927 03:58:47.956362 23519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:47.952369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.124:46615 (local address 127.0.0.1:47666)
0927 03:58:47.952731 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:47.952742 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:47.952798 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:47.953033 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:47.953034 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:47.953037 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:58:47.953113 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:47.953116 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:47.954078 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:47.954079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:47.955305 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:47.955308 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:47.956075 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:47.956076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:47.956141 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:47.956145 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:58:47.956206 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 03:58:47.956254 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":42,"threads_started":1}
I20260927 03:58:47.956367 23523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:47.952686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50668 (local address 127.15.185.125:36857)
0927 03:58:47.952901 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:47.952903 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:47.953033 (+ 130us) server_negotiation.cc:407] Connection header received
0927 03:58:47.953052 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:47.953052 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:47.953060 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:47.953118 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:58:47.953263 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:47.953984 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:47.955405 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:47.955967 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:47.956088 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:47.956103 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:47.956222 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 03:58:47.956261 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":50,"threads_started":1}
I20260927 03:58:47.956362 23521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:47.952566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.125:36857 (local address 127.0.0.1:50668)
0927 03:58:47.952929 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:47.952939 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:47.952984 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:47.953152 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:47.953153 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:47.953155 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:58:47.953196 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:47.953198 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:47.954073 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:47.954074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:47.955305 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:47.955308 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:47.956077 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:47.956078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:47.956141 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:47.956145 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:58:47.956206 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 03:58:47.956254 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":43,"threads_started":1}
I20260927 03:58:47.956602 23003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "87685032f24e44c59ad7c48aaff26803" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "08795fc9741d45d3953bb7183c4cb80f" is_pre_election: true
I20260927 03:58:47.956689 23003 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87685032f24e44c59ad7c48aaff26803 in term 1.
I20260927 03:58:47.956873 22811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "08795fc9741d45d3953bb7183c4cb80f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "87685032f24e44c59ad7c48aaff26803" is_pre_election: true
I20260927 03:58:47.956857 22787 leader_election.cc:302] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 08795fc9741d45d3953bb7183c4cb80f, 87685032f24e44c59ad7c48aaff26803; no voters: 1868152744144c8f97d2a16fc4d34438
I20260927 03:58:47.957005 22811 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08795fc9741d45d3953bb7183c4cb80f in term 1.
I20260927 03:58:47.957095 23516 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 03:58:47.957137 23516 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1868152744144c8f97d2a16fc4d34438)
I20260927 03:58:47.957158 23516 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803 [term 1 FOLLOWER]: Advancing to term 2
I20260927 03:58:47.957161 22978 leader_election.cc:302] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 08795fc9741d45d3953bb7183c4cb80f, 87685032f24e44c59ad7c48aaff26803; no voters: 1868152744144c8f97d2a16fc4d34438
I20260927 03:58:47.957254 23517 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 03:58:47.957298 23517 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1868152744144c8f97d2a16fc4d34438)
I20260927 03:58:47.957322 23517 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f [term 1 FOLLOWER]: Advancing to term 2
I20260927 03:58:47.957928 23517 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1868152744144c8f97d2a16fc4d34438" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 43909 } } peers { permanent_uuid: "87685032f24e44c59ad7c48aaff26803" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 36857 } attrs { promote: false } } peers { permanent_uuid: "08795fc9741d45d3953bb7183c4cb80f" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 46615 } attrs { promote: false } }
I20260927 03:58:47.957927 23516 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1868152744144c8f97d2a16fc4d34438" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 43909 } } peers { permanent_uuid: "87685032f24e44c59ad7c48aaff26803" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 36857 } attrs { promote: false } } peers { permanent_uuid: "08795fc9741d45d3953bb7183c4cb80f" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 46615 } attrs { promote: false } }
I20260927 03:58:47.958082 23517 leader_election.cc:288] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f [CANDIDATE]: Term 2 election: Requested vote from peers 1868152744144c8f97d2a16fc4d34438 (127.15.185.126:43909), 87685032f24e44c59ad7c48aaff26803 (127.15.185.125:36857)
I20260927 03:58:47.958091 23516 leader_election.cc:288] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803 [CANDIDATE]: Term 2 election: Requested vote from peers 1868152744144c8f97d2a16fc4d34438 (127.15.185.126:43909), 08795fc9741d45d3953bb7183c4cb80f (127.15.185.124:46615)
I20260927 03:58:47.958230 22811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "08795fc9741d45d3953bb7183c4cb80f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "87685032f24e44c59ad7c48aaff26803"
I20260927 03:58:47.958252 23521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:47.958151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:43909 (local address 127.0.0.1:38870)
0927 03:58:47.958193 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:47.958221 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:43909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260927 03:58:47.958309 22811 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 08795fc9741d45d3953bb7183c4cb80f in current term 2: Already voted for candidate 87685032f24e44c59ad7c48aaff26803 in this term.
I20260927 03:58:47.958326 23003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "87685032f24e44c59ad7c48aaff26803" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "08795fc9741d45d3953bb7183c4cb80f"
I20260927 03:58:47.958361 23519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:47.958205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:43909 (local address 127.0.0.1:38876)
0927 03:58:47.958290 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:47.958312 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:43909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260927 03:58:47.958392 23003 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 87685032f24e44c59ad7c48aaff26803 in current term 2: Already voted for candidate 08795fc9741d45d3953bb7183c4cb80f in this term.
I20260927 03:58:47.958503 23518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:47.958428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:43909 (local address 127.0.0.1:38886)
0927 03:58:47.958462 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:47.958482 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:43909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260927 03:58:47.958559 22979 leader_election.cc:334] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1868152744144c8f97d2a16fc4d34438 (127.15.185.126:43909): Network error: Client connection negotiation failed: client connection to 127.15.185.126:43909: connect: Connection refused (error 111)
I20260927 03:58:47.958628 22979 leader_election.cc:302] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08795fc9741d45d3953bb7183c4cb80f; no voters: 1868152744144c8f97d2a16fc4d34438, 87685032f24e44c59ad7c48aaff26803
I20260927 03:58:47.958705 23519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:47.958601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:43909 (local address 127.0.0.1:38892)
0927 03:58:47.958634 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:47.958654 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:43909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260927 03:58:47.958707 23517 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260927 03:58:47.958757 22786 leader_election.cc:334] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1868152744144c8f97d2a16fc4d34438 (127.15.185.126:43909): Network error: Client connection negotiation failed: client connection to 127.15.185.126:43909: connect: Connection refused (error 111)
I20260927 03:58:47.958791 22786 leader_election.cc:302] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87685032f24e44c59ad7c48aaff26803; no voters: 08795fc9741d45d3953bb7183c4cb80f, 1868152744144c8f97d2a16fc4d34438
I20260927 03:58:47.958865 23516 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 03:58:48.448565 23526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:48.448286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:43909 (local address 127.15.185.66:56765)
0927 03:58:48.448445 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:48.448487 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:43909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":80,"threads_started":1}
I20260927 03:58:48.449462 23527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:48.449261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:43909 (local address 127.15.185.67:48369)
0927 03:58:48.449379 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:48.449416 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:43909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":59,"threads_started":1}
I20260927 03:58:48.449558 23528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:48.449344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:43909 (local address 127.15.185.65:35235)
0927 03:58:48.449467 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:48.449508 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:43909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":54,"threads_started":1}
I20260927 03:58:49.449287 23529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:49.449002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:43909 (local address 127.15.185.66:42039)
0927 03:58:49.449158 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:49.449198 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:43909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":94,"threads_started":1}
I20260927 03:58:49.450002 23530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:49.449772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:43909 (local address 127.15.185.67:40261)
0927 03:58:49.449924 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:49.449957 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:43909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":57,"threads_started":1}
I20260927 03:58:49.450157 23531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:49.449956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:43909 (local address 127.15.185.65:45219)
0927 03:58:49.450081 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:49.450117 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:43909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":47,"threads_started":1}
I20260927 03:58:49.694793 23532 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:58:49.694885 23532 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1868152744144c8f97d2a16fc4d34438" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 43909 } } peers { permanent_uuid: "87685032f24e44c59ad7c48aaff26803" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 36857 } attrs { promote: false } } peers { permanent_uuid: "08795fc9741d45d3953bb7183c4cb80f" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 46615 } attrs { promote: false } }
I20260927 03:58:49.695051 23532 leader_election.cc:288] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1868152744144c8f97d2a16fc4d34438 (127.15.185.126:43909), 08795fc9741d45d3953bb7183c4cb80f (127.15.185.124:46615)
I20260927 03:58:49.695313 23003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "87685032f24e44c59ad7c48aaff26803" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "08795fc9741d45d3953bb7183c4cb80f" is_pre_election: true
I20260927 03:58:49.695390 23533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:49.695206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:43909 (local address 127.0.0.1:47184)
0927 03:58:49.695310 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:49.695352 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:43909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":62,"threads_started":1}
I20260927 03:58:49.695422 23003 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87685032f24e44c59ad7c48aaff26803 in term 2.
I20260927 03:58:49.695602 22787 leader_election.cc:302] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08795fc9741d45d3953bb7183c4cb80f, 87685032f24e44c59ad7c48aaff26803; no voters:
I20260927 03:58:49.695719 23532 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 03:58:49.695775 23532 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:58:49.695796 23532 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803 [term 2 FOLLOWER]: Advancing to term 3
I20260927 03:58:49.695837 23533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:49.695711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:43909 (local address 127.0.0.1:47194)
0927 03:58:49.695756 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:49.695809 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:43909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 03:58:49.695904 22786 leader_election.cc:334] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1868152744144c8f97d2a16fc4d34438 (127.15.185.126:43909): Network error: Client connection negotiation failed: client connection to 127.15.185.126:43909: connect: Connection refused (error 111)
I20260927 03:58:49.696566 23532 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1868152744144c8f97d2a16fc4d34438" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 43909 } } peers { permanent_uuid: "87685032f24e44c59ad7c48aaff26803" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 36857 } attrs { promote: false } } peers { permanent_uuid: "08795fc9741d45d3953bb7183c4cb80f" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 46615 } attrs { promote: false } }
I20260927 03:58:49.696713 23532 leader_election.cc:288] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803 [CANDIDATE]: Term 3 election: Requested vote from peers 1868152744144c8f97d2a16fc4d34438 (127.15.185.126:43909), 08795fc9741d45d3953bb7183c4cb80f (127.15.185.124:46615)
I20260927 03:58:49.696897 23533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:49.696814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:43909 (local address 127.0.0.1:47196)
0927 03:58:49.696846 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:49.696870 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:43909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260927 03:58:49.696893 23003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "87685032f24e44c59ad7c48aaff26803" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "08795fc9741d45d3953bb7183c4cb80f"
I20260927 03:58:49.696983 23003 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f [term 2 FOLLOWER]: Advancing to term 3
I20260927 03:58:49.697261 23533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:49.697164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:43909 (local address 127.0.0.1:47208)
0927 03:58:49.697216 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:49.697241 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:43909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 03:58:49.697359 22786 leader_election.cc:334] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 1868152744144c8f97d2a16fc4d34438 (127.15.185.126:43909): Network error: Client connection negotiation failed: client connection to 127.15.185.126:43909: connect: Connection refused (error 111)
I20260927 03:58:49.697733 23003 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87685032f24e44c59ad7c48aaff26803 in term 3.
I20260927 03:58:49.697893 22787 leader_election.cc:302] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 08795fc9741d45d3953bb7183c4cb80f, 87685032f24e44c59ad7c48aaff26803; no voters: 1868152744144c8f97d2a16fc4d34438
I20260927 03:58:49.698005 23532 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 03:58:49.698135 23532 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803 [term 3 LEADER]: Becoming Leader. State: Replica: 87685032f24e44c59ad7c48aaff26803, State: Running, Role: LEADER
I20260927 03:58:49.698211 23532 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1868152744144c8f97d2a16fc4d34438" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 43909 } } peers { permanent_uuid: "87685032f24e44c59ad7c48aaff26803" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 36857 } attrs { promote: false } } peers { permanent_uuid: "08795fc9741d45d3953bb7183c4cb80f" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 46615 } attrs { promote: false } }
I20260927 03:58:49.698549 23535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87685032f24e44c59ad7c48aaff26803. Latest consensus state: current_term: 3 leader_uuid: "87685032f24e44c59ad7c48aaff26803" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1868152744144c8f97d2a16fc4d34438" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 43909 } } peers { permanent_uuid: "87685032f24e44c59ad7c48aaff26803" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 36857 } attrs { promote: false } } peers { permanent_uuid: "08795fc9741d45d3953bb7183c4cb80f" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 46615 } attrs { promote: false } } }
I20260927 03:58:49.698648 23535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803 [sys.catalog]: This master's current role is: LEADER
I20260927 03:58:49.698819 23537 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:58:49.699275 23537 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:58:49.699331 23537 catalog_manager.cc:1259] Loaded cluster ID: 019b672ca57e471d9afa3c19afb55a10
I20260927 03:58:49.699348 23537 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:58:49.699479 23537 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:58:49.699530 23537 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803: Loaded TSK: 0
I20260927 03:58:49.699618 23537 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:58:49.719802 22801 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50656:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:58:49.721335 22801 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:58:49.722257 23533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:49.722162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:43909 (local address 127.0.0.1:47226)
0927 03:58:49.722196 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:49.722230 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:43909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260927 03:58:49.722311 23003 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f [term 3 FOLLOWER]: Refusing update from remote peer 87685032f24e44c59ad7c48aaff26803: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260927 03:58:49.722563 23532 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08795fc9741d45d3953bb7183c4cb80f" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 46615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260927 03:58:49.722625 23533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:49.722519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:43909 (local address 127.0.0.1:47228)
0927 03:58:49.722570 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:49.722602 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:43909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 03:58:49.722736 22786 consensus_peers.cc:487] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803 -> Peer 1868152744144c8f97d2a16fc4d34438 (127.15.185.126:43909): Couldn't send request to peer 1868152744144c8f97d2a16fc4d34438. Status: Network error: Client connection negotiation failed: client connection to 127.15.185.126:43909: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 03:58:49.723271 23539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87685032f24e44c59ad7c48aaff26803. Latest consensus state: current_term: 3 leader_uuid: "87685032f24e44c59ad7c48aaff26803" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1868152744144c8f97d2a16fc4d34438" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 43909 } } peers { permanent_uuid: "87685032f24e44c59ad7c48aaff26803" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 36857 } attrs { promote: false } } peers { permanent_uuid: "08795fc9741d45d3953bb7183c4cb80f" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 46615 } attrs { promote: false } } }
I20260927 03:58:49.723380 23539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:58:49.723806 23532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "87685032f24e44c59ad7c48aaff26803" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1868152744144c8f97d2a16fc4d34438" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 43909 } } peers { permanent_uuid: "87685032f24e44c59ad7c48aaff26803" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 36857 } attrs { promote: false } } peers { permanent_uuid: "08795fc9741d45d3953bb7183c4cb80f" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 46615 } attrs { promote: false } } }
I20260927 03:58:49.723901 23532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803 [sys.catalog]: This master's current role is: LEADER
I20260927 03:58:49.724002 23539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "87685032f24e44c59ad7c48aaff26803" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "1868152744144c8f97d2a16fc4d34438" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 43909 } } peers { permanent_uuid: "87685032f24e44c59ad7c48aaff26803" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 36857 } attrs { promote: false } } peers { permanent_uuid: "08795fc9741d45d3953bb7183c4cb80f" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 46615 } attrs { promote: false } } }
I20260927 03:58:49.724144 23539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08795fc9741d45d3953bb7183c4cb80f [sys.catalog]: This master's current role is: FOLLOWER
I20260927 03:58:49.728777 23548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:49.725728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33480 (local address 127.15.185.67:33539)
0927 03:58:49.725820 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:49.725822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:49.725832 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:58:49.725896 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:49.725897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:49.725934 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:49.725995 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:58:49.726167 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:49.726950 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:49.727934 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:49.728423 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:49.728488 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:49.728499 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:49.728657 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 03:58:49.728703 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70,"thread_start_us":43,"threads_started":1}
I20260927 03:58:49.728802 23533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:49.725500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.65:35459 (local address 127.0.0.1:60976)
0927 03:58:49.725536 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:49.725548 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:49.725596 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:49.725967 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:49.725968 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:49.725969 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:58:49.726024 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:49.726026 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:49.726784 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:49.726785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:49.727883 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:49.727885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:49.728555 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:49.728556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:49.728616 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:49.728620 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:58:49.728670 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 03:58:49.728708 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260927 03:58:49.728760 23547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:49.725567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.67:33539 (local address 127.0.0.1:33480)
0927 03:58:49.725785 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:49.725796 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:49.725843 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:49.726042 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:49.726043 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:49.726045 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:58:49.726089 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:49.726090 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:49.727038 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:49.727039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:49.727832 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:49.727834 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:49.728529 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:49.728529 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:49.728589 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:49.728592 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:58:49.728636 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 03:58:49.728674 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":2,"thread_start_us":67,"threads_started":1}
I20260927 03:58:49.728936 23545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:49.725602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60976 (local address 127.15.185.65:35459)
0927 03:58:49.725815 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:49.725817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:49.725826 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:58:49.725858 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:49.725859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:49.725870 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:49.725964 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:58:49.726109 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:49.726705 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:49.727979 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:49.728476 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:49.728556 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:49.728566 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:49.728790 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 03:58:49.728843 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":49,"threads_started":1}
I20260927 03:58:49.729137 23544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:49.725508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.66:40143 (local address 127.0.0.1:46140)
0927 03:58:49.726031 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:49.726050 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:49.726118 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:49.726282 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:49.726282 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:49.726284 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:58:49.726317 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:49.726318 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:49.727136 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:49.727137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:49.728206 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:49.728208 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:49.728957 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:49.728958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:49.729014 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:49.729017 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:58:49.729069 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 03:58:49.729096 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"mutex_wait_us":160,"thread_start_us":84,"threads_started":1}
I20260927 03:58:49.729190 23546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:49.725632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46140 (local address 127.15.185.66:40143)
0927 03:58:49.725738 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:49.725740 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:49.726095 (+ 355us) server_negotiation.cc:407] Connection header received
0927 03:58:49.726200 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:49.726201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:49.726213 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:49.726280 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:58:49.726370 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:49.727076 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:49.728283 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:49.728848 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:49.728914 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:49.728926 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:49.729080 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 03:58:49.729131 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79,"thread_start_us":51,"threads_started":1}
I20260927 03:58:49.729378 23239 tablet_service.cc:1467] Processing CreateTablet for tablet 7d7dd4eddc964230a89b6615abb5f068 (DEFAULT_TABLE table=test-table [id=df765558dde44c2987163c9678836f4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:58:49.729530 23239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d7dd4eddc964230a89b6615abb5f068. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:49.729622 23107 tablet_service.cc:1467] Processing CreateTablet for tablet 7d7dd4eddc964230a89b6615abb5f068 (DEFAULT_TABLE table=test-table [id=df765558dde44c2987163c9678836f4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:58:49.729707 23380 tablet_service.cc:1467] Processing CreateTablet for tablet 7d7dd4eddc964230a89b6615abb5f068 (DEFAULT_TABLE table=test-table [id=df765558dde44c2987163c9678836f4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:58:49.729746 23107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d7dd4eddc964230a89b6615abb5f068. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:49.729804 23380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d7dd4eddc964230a89b6615abb5f068. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:49.731110 23549 tablet_bootstrap.cc:492] T 7d7dd4eddc964230a89b6615abb5f068 P 55d1a890d9f4473fa298231905647c4a: Bootstrap starting.
I20260927 03:58:49.731423 23550 tablet_bootstrap.cc:492] T 7d7dd4eddc964230a89b6615abb5f068 P 20ea15da3bd74f2498765c4f9733de16: Bootstrap starting.
I20260927 03:58:49.731551 23549 tablet_bootstrap.cc:654] T 7d7dd4eddc964230a89b6615abb5f068 P 55d1a890d9f4473fa298231905647c4a: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:49.731781 23550 tablet_bootstrap.cc:654] T 7d7dd4eddc964230a89b6615abb5f068 P 20ea15da3bd74f2498765c4f9733de16: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:49.732065 23551 tablet_bootstrap.cc:492] T 7d7dd4eddc964230a89b6615abb5f068 P ce9010ea1dc94a0f91d6c8b10895c2f1: Bootstrap starting.
I20260927 03:58:49.732149 23549 tablet_bootstrap.cc:492] T 7d7dd4eddc964230a89b6615abb5f068 P 55d1a890d9f4473fa298231905647c4a: No bootstrap required, opened a new log
I20260927 03:58:49.732201 23549 ts_tablet_manager.cc:1397] T 7d7dd4eddc964230a89b6615abb5f068 P 55d1a890d9f4473fa298231905647c4a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:58:49.732331 23549 raft_consensus.cc:348] T 7d7dd4eddc964230a89b6615abb5f068 P 55d1a890d9f4473fa298231905647c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce9010ea1dc94a0f91d6c8b10895c2f1" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 35459 } } peers { permanent_uuid: "55d1a890d9f4473fa298231905647c4a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 40143 } } peers { permanent_uuid: "20ea15da3bd74f2498765c4f9733de16" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 33539 } }
I20260927 03:58:49.732385 23549 raft_consensus.cc:374] T 7d7dd4eddc964230a89b6615abb5f068 P 55d1a890d9f4473fa298231905647c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:49.732399 23549 raft_consensus.cc:729] T 7d7dd4eddc964230a89b6615abb5f068 P 55d1a890d9f4473fa298231905647c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55d1a890d9f4473fa298231905647c4a, State: Initialized, Role: FOLLOWER
I20260927 03:58:49.732436 23551 tablet_bootstrap.cc:654] T 7d7dd4eddc964230a89b6615abb5f068 P ce9010ea1dc94a0f91d6c8b10895c2f1: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:49.732455 23549 consensus_queue.cc:260] T 7d7dd4eddc964230a89b6615abb5f068 P 55d1a890d9f4473fa298231905647c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce9010ea1dc94a0f91d6c8b10895c2f1" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 35459 } } peers { permanent_uuid: "55d1a890d9f4473fa298231905647c4a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 40143 } } peers { permanent_uuid: "20ea15da3bd74f2498765c4f9733de16" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 33539 } }
I20260927 03:58:49.732620 23549 ts_tablet_manager.cc:1428] T 7d7dd4eddc964230a89b6615abb5f068 P 55d1a890d9f4473fa298231905647c4a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 03:58:49.732796 23550 tablet_bootstrap.cc:492] T 7d7dd4eddc964230a89b6615abb5f068 P 20ea15da3bd74f2498765c4f9733de16: No bootstrap required, opened a new log
I20260927 03:58:49.732873 23550 ts_tablet_manager.cc:1397] T 7d7dd4eddc964230a89b6615abb5f068 P 20ea15da3bd74f2498765c4f9733de16: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:58:49.732920 23529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:49.732794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:43909 (local address 127.15.185.66:38523)
0927 03:58:49.732862 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:49.732889 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:43909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 03:58:49.733012 23550 raft_consensus.cc:348] T 7d7dd4eddc964230a89b6615abb5f068 P 20ea15da3bd74f2498765c4f9733de16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce9010ea1dc94a0f91d6c8b10895c2f1" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 35459 } } peers { permanent_uuid: "55d1a890d9f4473fa298231905647c4a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 40143 } } peers { permanent_uuid: "20ea15da3bd74f2498765c4f9733de16" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 33539 } }
I20260927 03:58:49.733099 23550 raft_consensus.cc:374] T 7d7dd4eddc964230a89b6615abb5f068 P 20ea15da3bd74f2498765c4f9733de16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:49.733131 23550 raft_consensus.cc:729] T 7d7dd4eddc964230a89b6615abb5f068 P 20ea15da3bd74f2498765c4f9733de16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20ea15da3bd74f2498765c4f9733de16, State: Initialized, Role: FOLLOWER
I20260927 03:58:49.733223 23529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:49.733146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:43909 (local address 127.15.185.66:42553)
0927 03:58:49.733181 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:49.733202 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:43909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260927 03:58:49.733349 23551 tablet_bootstrap.cc:492] T 7d7dd4eddc964230a89b6615abb5f068 P ce9010ea1dc94a0f91d6c8b10895c2f1: No bootstrap required, opened a new log
I20260927 03:58:49.733413 23551 ts_tablet_manager.cc:1397] T 7d7dd4eddc964230a89b6615abb5f068 P ce9010ea1dc94a0f91d6c8b10895c2f1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:58:49.733537 23551 raft_consensus.cc:348] T 7d7dd4eddc964230a89b6615abb5f068 P ce9010ea1dc94a0f91d6c8b10895c2f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce9010ea1dc94a0f91d6c8b10895c2f1" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 35459 } } peers { permanent_uuid: "55d1a890d9f4473fa298231905647c4a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 40143 } } peers { permanent_uuid: "20ea15da3bd74f2498765c4f9733de16" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 33539 } }
I20260927 03:58:49.733609 23551 raft_consensus.cc:374] T 7d7dd4eddc964230a89b6615abb5f068 P ce9010ea1dc94a0f91d6c8b10895c2f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:49.733629 23551 raft_consensus.cc:729] T 7d7dd4eddc964230a89b6615abb5f068 P ce9010ea1dc94a0f91d6c8b10895c2f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce9010ea1dc94a0f91d6c8b10895c2f1, State: Initialized, Role: FOLLOWER
I20260927 03:58:49.733686 23551 consensus_queue.cc:260] T 7d7dd4eddc964230a89b6615abb5f068 P ce9010ea1dc94a0f91d6c8b10895c2f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce9010ea1dc94a0f91d6c8b10895c2f1" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 35459 } } peers { permanent_uuid: "55d1a890d9f4473fa298231905647c4a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 40143 } } peers { permanent_uuid: "20ea15da3bd74f2498765c4f9733de16" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 33539 } }
I20260927 03:58:49.733870 23551 ts_tablet_manager.cc:1428] T 7d7dd4eddc964230a89b6615abb5f068 P ce9010ea1dc94a0f91d6c8b10895c2f1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:49.734236 23531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:49.734127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:43909 (local address 127.15.185.65:54223)
0927 03:58:49.734178 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:49.734209 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:43909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 03:58:49.733955 23550 consensus_queue.cc:260] T 7d7dd4eddc964230a89b6615abb5f068 P 20ea15da3bd74f2498765c4f9733de16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce9010ea1dc94a0f91d6c8b10895c2f1" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 35459 } } peers { permanent_uuid: "55d1a890d9f4473fa298231905647c4a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 40143 } } peers { permanent_uuid: "20ea15da3bd74f2498765c4f9733de16" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 33539 } }
I20260927 03:58:49.734432 23550 ts_tablet_manager.cc:1428] T 7d7dd4eddc964230a89b6615abb5f068 P 20ea15da3bd74f2498765c4f9733de16: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260927 03:58:49.734583 23531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:49.734460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:43909 (local address 127.15.185.65:59507)
0927 03:58:49.734526 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:49.734561 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:43909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 03:58:49.734707 23530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:49.734615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:43909 (local address 127.15.185.67:36409)
0927 03:58:49.734652 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:49.734675 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:43909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260927 03:58:49.734973 23530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:49.734896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:43909 (local address 127.15.185.67:45127)
0927 03:58:49.734929 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:49.734951 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:43909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260927 03:58:49.985499 23554 raft_consensus.cc:482] T 7d7dd4eddc964230a89b6615abb5f068 P 55d1a890d9f4473fa298231905647c4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:58:49.985652 23554 raft_consensus.cc:504] T 7d7dd4eddc964230a89b6615abb5f068 P 55d1a890d9f4473fa298231905647c4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce9010ea1dc94a0f91d6c8b10895c2f1" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 35459 } } peers { permanent_uuid: "55d1a890d9f4473fa298231905647c4a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 40143 } } peers { permanent_uuid: "20ea15da3bd74f2498765c4f9733de16" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 33539 } }
I20260927 03:58:49.986042 23554 leader_election.cc:288] T 7d7dd4eddc964230a89b6615abb5f068 P 55d1a890d9f4473fa298231905647c4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce9010ea1dc94a0f91d6c8b10895c2f1 (127.15.185.65:35459), 20ea15da3bd74f2498765c4f9733de16 (127.15.185.67:33539)
I20260927 03:58:49.990070 23548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:49.986340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:47531 (local address 127.15.185.67:33539)
0927 03:58:49.986387 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:49.986389 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:49.986561 (+ 172us) server_negotiation.cc:407] Connection header received
0927 03:58:49.986634 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:49.986636 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:49.986646 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:49.986703 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:58:49.986842 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:49.987582 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:49.989154 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:49.989696 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:49.989773 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:49.989783 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:49.989954 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 03:58:49.989990 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 03:58:49.990058 23545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:49.986307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:57187 (local address 127.15.185.65:35459)
0927 03:58:49.986354 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:49.986355 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:49.986365 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:58:49.986462 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:49.986463 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:49.986472 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:49.986591 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:58:49.986786 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:49.987593 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:49.989154 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:49.989697 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:49.989773 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:49.989783 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:49.989954 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 03:58:49.989989 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 03:58:49.990104 23529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:49.986241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.65:35459 (local address 127.15.185.66:57187)
0927 03:58:49.986296 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:49.986307 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:49.986361 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:49.986616 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:49.986617 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:49.986620 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:58:49.986706 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:49.986708 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:49.987676 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:49.987676 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:49.989062 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:49.989065 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:49.989792 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:49.989793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:49.989868 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:49.989873 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:58:49.989930 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 03:58:49.989976 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":87}
I20260927 03:58:49.990119 23558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:49.986255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.67:33539 (local address 127.15.185.66:47531)
0927 03:58:49.986448 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:49.986471 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:49.986563 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:49.986717 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:49.986717 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:49.986719 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:58:49.986770 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:49.986772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:49.987673 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:49.987675 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:49.989062 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:49.989065 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:49.989790 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:49.989792 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:49.989871 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:49.989873 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 03:58:49.989931 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 03:58:49.989976 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":81,"thread_start_us":129,"threads_started":1}
I20260927 03:58:49.990414 23127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7dd4eddc964230a89b6615abb5f068" candidate_uuid: "55d1a890d9f4473fa298231905647c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce9010ea1dc94a0f91d6c8b10895c2f1" is_pre_election: true
I20260927 03:58:49.990473 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7dd4eddc964230a89b6615abb5f068" candidate_uuid: "55d1a890d9f4473fa298231905647c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20ea15da3bd74f2498765c4f9733de16" is_pre_election: true
I20260927 03:58:49.990554 23400 raft_consensus.cc:2454] T 7d7dd4eddc964230a89b6615abb5f068 P 20ea15da3bd74f2498765c4f9733de16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55d1a890d9f4473fa298231905647c4a in term 0.
I20260927 03:58:49.990594 23127 raft_consensus.cc:2454] T 7d7dd4eddc964230a89b6615abb5f068 P ce9010ea1dc94a0f91d6c8b10895c2f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55d1a890d9f4473fa298231905647c4a in term 0.
I20260927 03:58:49.990744 23200 leader_election.cc:302] T 7d7dd4eddc964230a89b6615abb5f068 P 55d1a890d9f4473fa298231905647c4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20ea15da3bd74f2498765c4f9733de16, 55d1a890d9f4473fa298231905647c4a; no voters:
I20260927 03:58:49.990932 23554 raft_consensus.cc:2781] T 7d7dd4eddc964230a89b6615abb5f068 P 55d1a890d9f4473fa298231905647c4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:58:49.990979 23554 raft_consensus.cc:482] T 7d7dd4eddc964230a89b6615abb5f068 P 55d1a890d9f4473fa298231905647c4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:58:49.990998 23554 raft_consensus.cc:3037] T 7d7dd4eddc964230a89b6615abb5f068 P 55d1a890d9f4473fa298231905647c4a [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:49.991734 23554 raft_consensus.cc:504] T 7d7dd4eddc964230a89b6615abb5f068 P 55d1a890d9f4473fa298231905647c4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce9010ea1dc94a0f91d6c8b10895c2f1" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 35459 } } peers { permanent_uuid: "55d1a890d9f4473fa298231905647c4a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 40143 } } peers { permanent_uuid: "20ea15da3bd74f2498765c4f9733de16" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 33539 } }
I20260927 03:58:49.991882 23554 leader_election.cc:288] T 7d7dd4eddc964230a89b6615abb5f068 P 55d1a890d9f4473fa298231905647c4a [CANDIDATE]: Term 1 election: Requested vote from peers ce9010ea1dc94a0f91d6c8b10895c2f1 (127.15.185.65:35459), 20ea15da3bd74f2498765c4f9733de16 (127.15.185.67:33539)
I20260927 03:58:49.992007 23127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7dd4eddc964230a89b6615abb5f068" candidate_uuid: "55d1a890d9f4473fa298231905647c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce9010ea1dc94a0f91d6c8b10895c2f1"
I20260927 03:58:49.992036 23400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7dd4eddc964230a89b6615abb5f068" candidate_uuid: "55d1a890d9f4473fa298231905647c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20ea15da3bd74f2498765c4f9733de16"
I20260927 03:58:49.992089 23127 raft_consensus.cc:3037] T 7d7dd4eddc964230a89b6615abb5f068 P ce9010ea1dc94a0f91d6c8b10895c2f1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:49.992098 23400 raft_consensus.cc:3037] T 7d7dd4eddc964230a89b6615abb5f068 P 20ea15da3bd74f2498765c4f9733de16 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:49.992856 23400 raft_consensus.cc:2454] T 7d7dd4eddc964230a89b6615abb5f068 P 20ea15da3bd74f2498765c4f9733de16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55d1a890d9f4473fa298231905647c4a in term 1.
I20260927 03:58:49.992853 23127 raft_consensus.cc:2454] T 7d7dd4eddc964230a89b6615abb5f068 P ce9010ea1dc94a0f91d6c8b10895c2f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55d1a890d9f4473fa298231905647c4a in term 1.
I20260927 03:58:49.993055 23198 leader_election.cc:302] T 7d7dd4eddc964230a89b6615abb5f068 P 55d1a890d9f4473fa298231905647c4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55d1a890d9f4473fa298231905647c4a, ce9010ea1dc94a0f91d6c8b10895c2f1; no voters:
I20260927 03:58:49.993188 23554 raft_consensus.cc:2781] T 7d7dd4eddc964230a89b6615abb5f068 P 55d1a890d9f4473fa298231905647c4a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:58:49.993371 23554 raft_consensus.cc:686] T 7d7dd4eddc964230a89b6615abb5f068 P 55d1a890d9f4473fa298231905647c4a [term 1 LEADER]: Becoming Leader. State: Replica: 55d1a890d9f4473fa298231905647c4a, State: Running, Role: LEADER
I20260927 03:58:49.993456 23554 consensus_queue.cc:237] T 7d7dd4eddc964230a89b6615abb5f068 P 55d1a890d9f4473fa298231905647c4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce9010ea1dc94a0f91d6c8b10895c2f1" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 35459 } } peers { permanent_uuid: "55d1a890d9f4473fa298231905647c4a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 40143 } } peers { permanent_uuid: "20ea15da3bd74f2498765c4f9733de16" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 33539 } }
I20260927 03:58:49.993525 23312 heartbeater.cc:503] Master 127.15.185.125:36857 was elected leader, sending a full tablet report...
I20260927 03:58:49.993906 23558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:49.993783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:43909 (local address 127.15.185.66:46399)
0927 03:58:49.993828 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:49.993856 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:43909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260927 03:58:49.994202 22801 catalog_manager.cc:5495] T 7d7dd4eddc964230a89b6615abb5f068 P 55d1a890d9f4473fa298231905647c4a reported cstate change: term changed from 0 to 1, leader changed from <none> to 55d1a890d9f4473fa298231905647c4a (127.15.185.66). New cstate: current_term: 1 leader_uuid: "55d1a890d9f4473fa298231905647c4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce9010ea1dc94a0f91d6c8b10895c2f1" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55d1a890d9f4473fa298231905647c4a" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 40143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20ea15da3bd74f2498765c4f9733de16" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 33539 } health_report { overall_health: UNKNOWN } } }
I20260927 03:58:50.011911 23546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:50.009368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46144 (local address 127.15.185.66:40143)
0927 03:58:50.009426 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:50.009429 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:50.009457 (+ 28us) server_negotiation.cc:407] Connection header received
0927 03:58:50.009501 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:50.009502 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:50.009519 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:50.009599 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:58:50.009747 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:50.010229 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:50.011003 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:50.011260 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:50.011350 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:50.011560 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:58:50.011669 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:58:50.011755 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:50.011796 (+ 41us) server_negotiation.cc:299] Negotiation successful
0927 03:58:50.011819 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 03:58:50.013232 23400 raft_consensus.cc:1261] T 7d7dd4eddc964230a89b6615abb5f068 P 20ea15da3bd74f2498765c4f9733de16 [term 1 FOLLOWER]: Refusing update from remote peer 55d1a890d9f4473fa298231905647c4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:58:50.013302 23127 raft_consensus.cc:1261] T 7d7dd4eddc964230a89b6615abb5f068 P ce9010ea1dc94a0f91d6c8b10895c2f1 [term 1 FOLLOWER]: Refusing update from remote peer 55d1a890d9f4473fa298231905647c4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:58:50.013515 23559 consensus_queue.cc:1035] T 7d7dd4eddc964230a89b6615abb5f068 P 55d1a890d9f4473fa298231905647c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "20ea15da3bd74f2498765c4f9733de16" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:58:50.013641 23554 consensus_queue.cc:1035] T 7d7dd4eddc964230a89b6615abb5f068 P 55d1a890d9f4473fa298231905647c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce9010ea1dc94a0f91d6c8b10895c2f1" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:58:50.014201 23175 heartbeater.cc:503] Master 127.15.185.125:36857 was elected leader, sending a full tablet report...
I20260927 03:58:50.014501 23531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:50.014378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:43909 (local address 127.15.185.65:54133)
0927 03:58:50.014434 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:50.014469 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:43909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 03:58:50.014638 23448 heartbeater.cc:503] Master 127.15.185.125:36857 was elected leader, sending a full tablet report...
I20260927 03:58:50.014715 23530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:50.014548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:43909 (local address 127.15.185.67:52651)
0927 03:58:50.014643 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:50.014680 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:43909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260927 03:58:50.016327 23531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:50.016225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:43909 (local address 127.15.185.65:51285)
0927 03:58:50.016273 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:50.016297 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:43909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 03:58:50.016654 23530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:50.016571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:43909 (local address 127.15.185.67:59007)
0927 03:58:50.016605 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:50.016627 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:43909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260927 03:58:50.016856 23558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:50.016038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:43909 (local address 127.15.185.66:57867)
0927 03:58:50.016781 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:50.016811 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:43909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":724}
I20260927 03:58:50.018831 22800 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:58:50.018918 22800 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:58:50.020155 22800 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 87685032f24e44c59ad7c48aaff26803: Sending DeleteTablet for 3 replicas of tablet 7d7dd4eddc964230a89b6615abb5f068
I20260927 03:58:50.020463 23380 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d7dd4eddc964230a89b6615abb5f068 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:58:50 UTC) from {username='oryx', principal='oryx/127.15.185.125@KRBTEST.COM'} at 127.0.0.1:33480
I20260927 03:58:50.020471 23239 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d7dd4eddc964230a89b6615abb5f068 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:58:50 UTC) from {username='oryx', principal='oryx/127.15.185.125@KRBTEST.COM'} at 127.0.0.1:46140
I20260927 03:58:50.020605 23107 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d7dd4eddc964230a89b6615abb5f068 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:58:50 UTC) from {username='oryx', principal='oryx/127.15.185.125@KRBTEST.COM'} at 127.0.0.1:60976
I20260927 03:58:50.020709 23572 tablet_replica.cc:330] T 7d7dd4eddc964230a89b6615abb5f068 P 55d1a890d9f4473fa298231905647c4a: stopping tablet replica
I20260927 03:58:50.020758 23572 raft_consensus.cc:2229] T 7d7dd4eddc964230a89b6615abb5f068 P 55d1a890d9f4473fa298231905647c4a [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:58:50.020768 23574 tablet_replica.cc:330] T 7d7dd4eddc964230a89b6615abb5f068 P ce9010ea1dc94a0f91d6c8b10895c2f1: stopping tablet replica
I20260927 03:58:50.020819 23574 raft_consensus.cc:2229] T 7d7dd4eddc964230a89b6615abb5f068 P ce9010ea1dc94a0f91d6c8b10895c2f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:58:50.020839 23572 raft_consensus.cc:2258] T 7d7dd4eddc964230a89b6615abb5f068 P 55d1a890d9f4473fa298231905647c4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:58:50.020861 23574 raft_consensus.cc:2258] T 7d7dd4eddc964230a89b6615abb5f068 P ce9010ea1dc94a0f91d6c8b10895c2f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:58:50.020936 23573 tablet_replica.cc:330] T 7d7dd4eddc964230a89b6615abb5f068 P 20ea15da3bd74f2498765c4f9733de16: stopping tablet replica
I20260927 03:58:50.020982 23573 raft_consensus.cc:2229] T 7d7dd4eddc964230a89b6615abb5f068 P 20ea15da3bd74f2498765c4f9733de16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:58:50.021020 23573 raft_consensus.cc:2258] T 7d7dd4eddc964230a89b6615abb5f068 P 20ea15da3bd74f2498765c4f9733de16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:58:50.021157 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 22952
I20260927 03:58:50.021266 23574 ts_tablet_manager.cc:1905] T 7d7dd4eddc964230a89b6615abb5f068 P ce9010ea1dc94a0f91d6c8b10895c2f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:58:50.021858 23573 ts_tablet_manager.cc:1905] T 7d7dd4eddc964230a89b6615abb5f068 P 20ea15da3bd74f2498765c4f9733de16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:58:50.023321 23572 ts_tablet_manager.cc:1905] T 7d7dd4eddc964230a89b6615abb5f068 P 55d1a890d9f4473fa298231905647c4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:58:50.024936 23573 ts_tablet_manager.cc:1918] T 7d7dd4eddc964230a89b6615abb5f068 P 20ea15da3bd74f2498765c4f9733de16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:58:50.025032 23573 log.cc:1196] T 7d7dd4eddc964230a89b6615abb5f068 P 20ea15da3bd74f2498765c4f9733de16: Deleting WAL directory at /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-2/wal/wals/7d7dd4eddc964230a89b6615abb5f068
I20260927 03:58:50.025039 23572 ts_tablet_manager.cc:1918] T 7d7dd4eddc964230a89b6615abb5f068 P 55d1a890d9f4473fa298231905647c4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:58:50.025141 23572 log.cc:1196] T 7d7dd4eddc964230a89b6615abb5f068 P 55d1a890d9f4473fa298231905647c4a: Deleting WAL directory at /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481498018407-16101-0/minicluster-data/ts-1/wal/wals/7d7dd4eddc964230a89b6615abb5f068
I20260927 03:58:50.025286 23573 ts_tablet_manager.cc:1939] T 7d7dd4eddc964230a89b6615abb5f068 P 20ea15da3bd74f2498765c4f9733de16: Deleting consensus metadata
I20260927 03:58:50.025409 23572 ts_tablet_manager.cc:1939] T 7d7dd4eddc964230a89b6615abb5f068 P 55d1a890d9f4473fa298231905647c4a: Deleting consensus metadata
I20260927 03:58:50.025637 22788 catalog_manager.cc:4841] TS 20ea15da3bd74f2498765c4f9733de16 (127.15.185.67:33539): tablet 7d7dd4eddc964230a89b6615abb5f068 (table test-table [id=df765558dde44c2987163c9678836f4d]) successfully deleted
I20260927 03:58:50.025736 22787 catalog_manager.cc:4841] TS 55d1a890d9f4473fa298231905647c4a (127.15.185.66:40143): tablet 7d7dd4eddc964230a89b6615abb5f068 (table test-table [id=df765558dde44c2987163c9678836f4d]) successfully deleted
W20260927 03:58:50.029088 23544 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:58:50.028938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.65:35459 (local address 127.0.0.1:60982)
0927 03:58:50.028981 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:50.028992 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:50.029033 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.65:35459: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":26}
W20260927 03:58:50.029213 22786 catalog_manager.cc:4584] TS ce9010ea1dc94a0f91d6c8b10895c2f1 (127.15.185.65:35459): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7d7dd4eddc964230a89b6615abb5f068: Network error: Client connection negotiation failed: client connection to 127.15.185.65:35459: BlockingWrite error: write error: Connection reset by peer (error 104)
I20260927 03:58:50.029366 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 23181
I20260927 03:58:50.033746 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 23317
I20260927 03:58:50.038190 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 22667
I20260927 03:58:50.042973 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 22855
2026-09-27T03:58:50Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6890 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:58:50 dist-test-slave-glgz krb5kdc[23577](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:58:50 dist-test-slave-glgz krb5kdc[23577](info): set up 2 sockets
Sep 27 03:58:50 dist-test-slave-glgz krb5kdc[23577](info): commencing operation
krb5kdc: starting...
W20260927 03:58:51.131927 16101 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Sep 27 03:58:51 dist-test-slave-glgz krb5kdc[23577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:58:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:58:51Z Disabled control of system clock
WARNING: no policy specified for oryx/127.15.185.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.185.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.185.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/krb5kdc/oryx_127.15.185.126.keytab.
Entry for principal oryx/127.15.185.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/krb5kdc/oryx_127.15.185.126.keytab.
I20260927 03:58:51.258131 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:34491
--webserver_interface=127.15.185.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/krb5kdc/oryx_127.15.185.126.keytab
--principal=oryx/127.15.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:42347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.185.126:34491
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:58:51.340996 23590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:51.341250 23590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:51.341286 23590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:51.342456 23590 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:58:51.342535 23590 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 03:58:51.342561 23590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:51.342588 23590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:51.342620 23590 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:58:51.342643 23590 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:58:51.344143 23590 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:42347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.185.126:34491
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/krb5kdc/oryx_127.15.185.126.keytab
--principal=oryx/127.15.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:34491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:51.344427 23590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:51.344667 23590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:51.349447 23596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:51.349475 23598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:51.349498 23595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:51.349615 23590 server_base.cc:1034] running on GCE node
I20260927 03:58:51.350003 23590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:51.350301 23590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:51.351445 23590 hybrid_clock.cc:648] HybridClock initialized: now 1790481531351423 us; error 43 us; skew 500 ppm
Sep 27 03:58:51 dist-test-slave-glgz krb5kdc[23577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481531, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:51.353955 23590 init.cc:377] Logged in from keytab as oryx/127.15.185.126@KRBTEST.COM (short username oryx)
I20260927 03:58:51.354351 23590 webserver.cc:460] Webserver started at http://127.15.185.126:42997/ using document root <none> and password file <none>
I20260927 03:58:51.354687 23590 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:51.354736 23590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:51.354876 23590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:51.356081 23590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/master-0/data/instance:
uuid: "523c89766816419d81500ea2272b97f5"
format_stamp: "Formatted at 2026-09-27 03:58:51 on dist-test-slave-glgz"
server_key: "022f000eae0e6f938fb211e0edb5a4b0"
server_key_iv: "7da48b161c6f2e74a58928ac4e8ef77a"
server_key_version: "encryptionkey@0"
I20260927 03:58:51.356443 23590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/master-0/wal/instance:
uuid: "523c89766816419d81500ea2272b97f5"
format_stamp: "Formatted at 2026-09-27 03:58:51 on dist-test-slave-glgz"
server_key: "022f000eae0e6f938fb211e0edb5a4b0"
server_key_iv: "7da48b161c6f2e74a58928ac4e8ef77a"
server_key_version: "encryptionkey@0"
I20260927 03:58:51.357752 23590 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:58:51.358510 23605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:51.358735 23590 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:58:51.358852 23590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/master-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/master-0/wal
uuid: "523c89766816419d81500ea2272b97f5"
format_stamp: "Formatted at 2026-09-27 03:58:51 on dist-test-slave-glgz"
server_key: "022f000eae0e6f938fb211e0edb5a4b0"
server_key_iv: "7da48b161c6f2e74a58928ac4e8ef77a"
server_key_version: "encryptionkey@0"
I20260927 03:58:51.358933 23590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:51.368029 23590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:51.370069 23590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:51.370193 23590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:51.374644 23590 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.126:34491
I20260927 03:58:51.374751 23667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.126:34491 every 8 connection(s)
I20260927 03:58:51.375089 23590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
I20260927 03:58:51.375734 23668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:51.377995 23668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 523c89766816419d81500ea2272b97f5: Bootstrap starting.
I20260927 03:58:51.378589 23668 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 523c89766816419d81500ea2272b97f5: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:51.378891 23668 log.cc:824] T 00000000000000000000000000000000 P 523c89766816419d81500ea2272b97f5: Log is configured to *not* fsync() on all Append() calls
I20260927 03:58:51.379587 23668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 523c89766816419d81500ea2272b97f5: No bootstrap required, opened a new log
I20260927 03:58:51.381542 23668 raft_consensus.cc:348] T 00000000000000000000000000000000 P 523c89766816419d81500ea2272b97f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "523c89766816419d81500ea2272b97f5" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 34491 } }
I20260927 03:58:51.381675 23668 raft_consensus.cc:374] T 00000000000000000000000000000000 P 523c89766816419d81500ea2272b97f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:51.381696 23668 raft_consensus.cc:729] T 00000000000000000000000000000000 P 523c89766816419d81500ea2272b97f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 523c89766816419d81500ea2272b97f5, State: Initialized, Role: FOLLOWER
I20260927 03:58:51.381860 23668 consensus_queue.cc:260] T 00000000000000000000000000000000 P 523c89766816419d81500ea2272b97f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "523c89766816419d81500ea2272b97f5" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 34491 } }
I20260927 03:58:51.381929 23668 raft_consensus.cc:388] T 00000000000000000000000000000000 P 523c89766816419d81500ea2272b97f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:58:51.381955 23668 raft_consensus.cc:482] T 00000000000000000000000000000000 P 523c89766816419d81500ea2272b97f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:58:51.381989 23668 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 523c89766816419d81500ea2272b97f5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:51.382542 23668 raft_consensus.cc:504] T 00000000000000000000000000000000 P 523c89766816419d81500ea2272b97f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "523c89766816419d81500ea2272b97f5" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 34491 } }
I20260927 03:58:51.382627 23668 leader_election.cc:302] T 00000000000000000000000000000000 P 523c89766816419d81500ea2272b97f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 523c89766816419d81500ea2272b97f5; no voters:
I20260927 03:58:51.382836 23668 leader_election.cc:288] T 00000000000000000000000000000000 P 523c89766816419d81500ea2272b97f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:58:51.382862 23671 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 523c89766816419d81500ea2272b97f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:58:51.382972 23671 raft_consensus.cc:686] T 00000000000000000000000000000000 P 523c89766816419d81500ea2272b97f5 [term 1 LEADER]: Becoming Leader. State: Replica: 523c89766816419d81500ea2272b97f5, State: Running, Role: LEADER
I20260927 03:58:51.382861 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 23590
I20260927 03:58:51.383116 23668 sys_catalog.cc:564] T 00000000000000000000000000000000 P 523c89766816419d81500ea2272b97f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:58:51.383152 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/master-0/wal/instance
I20260927 03:58:51.383111 23671 consensus_queue.cc:237] T 00000000000000000000000000000000 P 523c89766816419d81500ea2272b97f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "523c89766816419d81500ea2272b97f5" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 34491 } }
I20260927 03:58:51.383383 16101 external_mini_cluster.cc:1428] Setting key 28052a24842445b9a5983bcac79f8e9a
I20260927 03:58:51.384258 23672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 523c89766816419d81500ea2272b97f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "523c89766816419d81500ea2272b97f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "523c89766816419d81500ea2272b97f5" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 34491 } } }
I20260927 03:58:51.384344 23672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 523c89766816419d81500ea2272b97f5 [sys.catalog]: This master's current role is: LEADER
I20260927 03:58:51.384505 23673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 523c89766816419d81500ea2272b97f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 523c89766816419d81500ea2272b97f5. Latest consensus state: current_term: 1 leader_uuid: "523c89766816419d81500ea2272b97f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "523c89766816419d81500ea2272b97f5" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 34491 } } }
I20260927 03:58:51.384600 23673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 523c89766816419d81500ea2272b97f5 [sys.catalog]: This master's current role is: LEADER
I20260927 03:58:51.384760 23685 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:58:51.385413 23685 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:58:51.386901 23685 catalog_manager.cc:1347] Generated new cluster ID: 12df0986e04449469aa83423c29aa5a1
I20260927 03:58:51.386956 23685 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 03:58:51 dist-test-slave-glgz krb5kdc[23577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM
I20260927 03:58:51.395192 23681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:51.384004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57206 (local address 127.15.185.126:34491)
0927 03:58:51.384171 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:51.384180 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:51.384210 (+ 30us) server_negotiation.cc:407] Connection header received
0927 03:58:51.384501 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:51.384509 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:51.384588 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:51.384743 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:51.385531 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:51.386880 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:51.387977 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:51.388598 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:51.388697 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:51.391537 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:51.391541 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:51.391541 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:51.391554 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:51.393450 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:51.393864 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:51.393867 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:51.393867 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:51.393969 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:51.394190 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:51.394190 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:51.394191 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:51.394482 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:51.394626 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:51.394744 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 03:58:51.394870 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":61,"threads_started":1}
I20260927 03:58:51.414620 23685 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:58:51.415427 23685 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:58:51.427203 23685 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 523c89766816419d81500ea2272b97f5: Generated new TSK 0
I20260927 03:58:51.427470 23685 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.15.185.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.185.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.185.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/krb5kdc/oryx_127.15.185.65.keytab.
Entry for principal oryx/127.15.185.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/krb5kdc/oryx_127.15.185.65.keytab.
I20260927 03:58:51.458717 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.65:0
--local_ip_for_outbound_sockets=127.15.185.65
--webserver_interface=127.15.185.65
--webserver_port=0
--tserver_master_addrs=127.15.185.126:34491
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/krb5kdc/oryx_127.15.185.65.keytab
--principal=oryx/127.15.185.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:42347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:58:51.539821 23694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:51.540035 23694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:51.540059 23694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:51.541126 23694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:51.541162 23694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:51.541181 23694 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 03:58:51.541199 23694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.65
I20260927 03:58:51.542598 23694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:42347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/krb5kdc/oryx_127.15.185.65.keytab
--principal=oryx/127.15.185.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.185.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.185.126:34491
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.65
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:51.542806 23694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:51.543026 23694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:51.547981 23702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:51.547973 23699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:51.547973 23700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:51.548111 23694 server_base.cc:1034] running on GCE node
I20260927 03:58:51.548386 23694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:51.548588 23694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:51.549752 23694 hybrid_clock.cc:648] HybridClock initialized: now 1790481531549737 us; error 27 us; skew 500 ppm
Sep 27 03:58:51 dist-test-slave-glgz krb5kdc[23577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481531, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.185.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:51.552443 23694 init.cc:377] Logged in from keytab as oryx/127.15.185.65@KRBTEST.COM (short username oryx)
I20260927 03:58:51.552980 23694 webserver.cc:460] Webserver started at http://127.15.185.65:33719/ using document root <none> and password file <none>
I20260927 03:58:51.553220 23694 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:51.553295 23694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:51.553469 23694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:51.554605 23694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/ts-0/data/instance:
uuid: "983a3b316b1f4809ae803c17f6c0def5"
format_stamp: "Formatted at 2026-09-27 03:58:51 on dist-test-slave-glgz"
server_key: "096ea2375158b2bcd91ff33fa8a5f253"
server_key_iv: "6b9da36cd111f48f9831633165b9e466"
server_key_version: "encryptionkey@0"
I20260927 03:58:51.555176 23694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/ts-0/wal/instance:
uuid: "983a3b316b1f4809ae803c17f6c0def5"
format_stamp: "Formatted at 2026-09-27 03:58:51 on dist-test-slave-glgz"
server_key: "096ea2375158b2bcd91ff33fa8a5f253"
server_key_iv: "6b9da36cd111f48f9831633165b9e466"
server_key_version: "encryptionkey@0"
I20260927 03:58:51.556638 23694 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:58:51.557443 23709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:51.557634 23694 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:58:51.557741 23694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/ts-0/wal
uuid: "983a3b316b1f4809ae803c17f6c0def5"
format_stamp: "Formatted at 2026-09-27 03:58:51 on dist-test-slave-glgz"
server_key: "096ea2375158b2bcd91ff33fa8a5f253"
server_key_iv: "6b9da36cd111f48f9831633165b9e466"
server_key_version: "encryptionkey@0"
I20260927 03:58:51.557834 23694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:51.574715 23694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:51.576947 23694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:51.577080 23694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:51.577771 23694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:58:51.577826 23694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:51.577860 23694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:58:51.577886 23694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:51.584929 23694 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.65:46859
I20260927 03:58:51.585014 23824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.65:46859 every 8 connection(s)
I20260927 03:58:51.585330 23694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
Sep 27 03:58:51 dist-test-slave-glgz krb5kdc[23577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481531, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.185.65@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM
I20260927 03:58:51.589169 23681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:51.578194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:39923 (local address 127.15.185.126:34491)
0927 03:58:51.578267 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:51.578270 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:51.578518 (+ 248us) server_negotiation.cc:407] Connection header received
0927 03:58:51.579112 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:51.579114 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:51.579128 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:51.579198 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:51.580130 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:51.581012 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:51.582859 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:51.583220 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:51.583332 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:51.586081 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:51.586083 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:51.586084 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:51.586092 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:51.587593 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:51.588095 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:51.588097 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:51.588098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:51.588164 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:51.588454 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:51.588455 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:51.588456 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:51.588586 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:51.588660 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:51.588971 (+ 311us) server_negotiation.cc:299] Negotiation successful
0927 03:58:51.589066 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 03:58:51.589392 23718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:51.578200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34491 (local address 127.15.185.65:39923)
0927 03:58:51.578376 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:51.578422 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:51.578973 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:51.579267 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:51.579270 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:51.579693 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:51.580031 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:51.580038 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:51.581125 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:51.581127 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:51.582751 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:51.582755 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:51.583341 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:51.583342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:51.583507 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:51.584181 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:51.584193 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:51.585968 (+ 1775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:51.587706 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:51.587714 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:51.587717 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:51.587989 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:51.588245 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:51.588245 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:51.588246 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:51.588372 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:51.588664 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:51.588671 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:51.588823 (+ 152us) client_negotiation.cc:769] Sending connection context
0927 03:58:51.588948 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 03:58:51.589126 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":80,"threads_started":1}
I20260927 03:58:51.589959 23825 heartbeater.cc:347] Connected to a master server at 127.15.185.126:34491
I20260927 03:58:51.590070 23825 heartbeater.cc:464] Registering TS with master...
I20260927 03:58:51.590257 23825 heartbeater.cc:511] Master 127.15.185.126:34491 requested a full tablet report, sending...
I20260927 03:58:51.590842 23621 ts_manager.cc:194] Registered new tserver with Master: 983a3b316b1f4809ae803c17f6c0def5 (127.15.185.65:46859)
I20260927 03:58:51.591586 23621 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.185.65@KRBTEST.COM'} at 127.15.185.65:39923
I20260927 03:58:51.593757 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 23694
I20260927 03:58:51.593899 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/ts-0/wal/instance
I20260927 03:58:51.594159 16101 external_mini_cluster.cc:1428] Setting key 2344881d7b729896f335d915828fd879
I20260927 03:58:51.597133 23831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:51.591036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34491 (local address 127.15.185.65:35167)
0927 03:58:51.591200 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:51.591218 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:51.591300 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:51.591441 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:51.591442 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:51.591686 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:51.591754 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:51.591756 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:51.592480 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:51.592480 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:51.593021 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:51.593023 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:51.593381 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:51.593382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:51.593440 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:51.594044 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:51.594051 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:51.594433 (+ 382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:51.596054 (+ 1621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:51.596055 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:51.596056 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:51.596288 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:51.596520 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:51.596520 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:51.596521 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:51.596595 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:51.596892 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:51.596892 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:51.596929 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 03:58:51.596981 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 03:58:51.597062 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":78,"threads_started":1}
I20260927 03:58:51.597213 23681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:51.591094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:35167 (local address 127.15.185.126:34491)
0927 03:58:51.591134 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:51.591136 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:51.591283 (+ 147us) server_negotiation.cc:407] Connection header received
0927 03:58:51.591363 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:51.591364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:51.591375 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:51.591436 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:51.591844 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:51.592347 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:51.593105 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:51.593283 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:51.593341 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:51.594526 (+ 1185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:51.594528 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:51.594528 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:51.594534 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:51.595984 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:51.596377 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:51.596378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:51.596379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:51.596446 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:51.596659 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:51.596659 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:51.596660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:51.596795 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:51.596877 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:51.597027 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 03:58:51.597117 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
WARNING: no policy specified for oryx/127.15.185.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.185.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.185.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/krb5kdc/oryx_127.15.185.66.keytab.
Entry for principal oryx/127.15.185.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/krb5kdc/oryx_127.15.185.66.keytab.
I20260927 03:58:51.622455 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.66:0
--local_ip_for_outbound_sockets=127.15.185.66
--webserver_interface=127.15.185.66
--webserver_port=0
--tserver_master_addrs=127.15.185.126:34491
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/krb5kdc/oryx_127.15.185.66.keytab
--principal=oryx/127.15.185.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:42347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:58:51.703478 23834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:51.703735 23834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:51.703775 23834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:51.704934 23834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:51.705011 23834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:51.705045 23834 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 03:58:51.705085 23834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.66
I20260927 03:58:51.706590 23834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:42347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/krb5kdc/oryx_127.15.185.66.keytab
--principal=oryx/127.15.185.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.185.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.185.126:34491
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.66
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:51.706875 23834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:51.707158 23834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:51.712273 23840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:51.712285 23842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:51.712330 23839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:51.712576 23834 server_base.cc:1034] running on GCE node
I20260927 03:58:51.712811 23834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:51.713074 23834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:51.714241 23834 hybrid_clock.cc:648] HybridClock initialized: now 1790481531714237 us; error 40 us; skew 500 ppm
Sep 27 03:58:51 dist-test-slave-glgz krb5kdc[23577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481531, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.185.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:51.716919 23834 init.cc:377] Logged in from keytab as oryx/127.15.185.66@KRBTEST.COM (short username oryx)
I20260927 03:58:51.717310 23834 webserver.cc:460] Webserver started at http://127.15.185.66:37341/ using document root <none> and password file <none>
I20260927 03:58:51.717597 23834 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:51.717669 23834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:51.717825 23834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:51.718904 23834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/ts-1/data/instance:
uuid: "40c4cc7b32f14b0e89a6d4e89511a047"
format_stamp: "Formatted at 2026-09-27 03:58:51 on dist-test-slave-glgz"
server_key: "382103c8b87a53fbf248d7a9ad47c514"
server_key_iv: "d666dcdc313ac137a9f266d96cf8ac81"
server_key_version: "encryptionkey@0"
I20260927 03:58:51.719290 23834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/ts-1/wal/instance:
uuid: "40c4cc7b32f14b0e89a6d4e89511a047"
format_stamp: "Formatted at 2026-09-27 03:58:51 on dist-test-slave-glgz"
server_key: "382103c8b87a53fbf248d7a9ad47c514"
server_key_iv: "d666dcdc313ac137a9f266d96cf8ac81"
server_key_version: "encryptionkey@0"
I20260927 03:58:51.720667 23834 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:58:51.721433 23849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:51.721715 23834 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:58:51.721827 23834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/ts-1/wal
uuid: "40c4cc7b32f14b0e89a6d4e89511a047"
format_stamp: "Formatted at 2026-09-27 03:58:51 on dist-test-slave-glgz"
server_key: "382103c8b87a53fbf248d7a9ad47c514"
server_key_iv: "d666dcdc313ac137a9f266d96cf8ac81"
server_key_version: "encryptionkey@0"
I20260927 03:58:51.721903 23834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:51.743418 23834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:51.745868 23834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:51.746044 23834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:51.746589 23834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:58:51.746655 23834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:51.746692 23834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:58:51.746723 23834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 03:58:51 dist-test-slave-glgz krb5kdc[23577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481531, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.185.66@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM
I20260927 03:58:51.754140 23834 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.66:38731
I20260927 03:58:51.754225 23964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.66:38731 every 8 connection(s)
I20260927 03:58:51.754571 23834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
I20260927 03:58:51.757314 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 23834
I20260927 03:58:51.757417 23681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:51.747112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:34933 (local address 127.15.185.126:34491)
0927 03:58:51.747169 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:51.747171 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:51.747555 (+ 384us) server_negotiation.cc:407] Connection header received
0927 03:58:51.748050 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:51.748051 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:51.748065 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:51.748144 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:51.748868 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:51.749359 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:51.750408 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:51.750653 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:51.750736 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:51.754070 (+ 3334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:51.754071 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:51.754072 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:51.754080 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:51.755721 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:51.756304 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:51.756305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:51.756306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:51.756375 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:51.756688 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:51.756689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:51.756690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:51.756885 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:51.756945 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:51.757236 (+ 291us) server_negotiation.cc:299] Negotiation successful
0927 03:58:51.757317 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 03:58:51.757436 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/ts-1/wal/instance
I20260927 03:58:51.757611 23858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:51.747199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34491 (local address 127.15.185.66:34933)
0927 03:58:51.747388 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:51.747453 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:51.747940 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:51.748245 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:51.748248 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:51.748553 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:51.748761 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:51.748766 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:51.749459 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:51.749460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:51.750307 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:51.750309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:51.750745 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:51.750747 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:51.750878 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:51.751469 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:51.751481 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:51.753979 (+ 2498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:51.755794 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:51.755797 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:51.755799 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:51.756218 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:51.756442 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:51.756443 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:51.756443 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:51.756595 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:51.756963 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:51.756969 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:51.757103 (+ 134us) client_negotiation.cc:769] Sending connection context
0927 03:58:51.757214 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 03:58:51.757374 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":87,"threads_started":1}
I20260927 03:58:51.757663 16101 external_mini_cluster.cc:1428] Setting key 120b29e2925079d1d862fd83876def3e
I20260927 03:58:51.758136 23965 heartbeater.cc:347] Connected to a master server at 127.15.185.126:34491
I20260927 03:58:51.758225 23965 heartbeater.cc:464] Registering TS with master...
I20260927 03:58:51.758419 23965 heartbeater.cc:511] Master 127.15.185.126:34491 requested a full tablet report, sending...
I20260927 03:58:51.759173 23620 ts_manager.cc:194] Registered new tserver with Master: 40c4cc7b32f14b0e89a6d4e89511a047 (127.15.185.66:38731)
I20260927 03:58:51.759747 23620 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.185.66@KRBTEST.COM'} at 127.15.185.66:34933
WARNING: no policy specified for oryx/127.15.185.67@KRBTEST.COM; defaulting to no policy
I20260927 03:58:51.766609 23972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:51.760019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34491 (local address 127.15.185.66:50651)
0927 03:58:51.760179 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:51.760191 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:51.760254 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:51.760419 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:51.760420 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:51.760632 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:51.760727 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:51.760729 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:51.761473 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:51.761474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:51.762389 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:51.762392 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:51.762860 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:51.762861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:51.762927 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:51.763481 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:51.763490 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:51.763780 (+ 290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:51.765476 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:51.765477 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:51.765477 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:51.765667 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:51.765890 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:51.765891 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:51.765891 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:51.765955 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:51.766252 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:51.766253 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:51.766300 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 03:58:51.766403 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 03:58:51.766511 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":68,"threads_started":1}
I20260927 03:58:51.766604 23681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:51.760075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:50651 (local address 127.15.185.126:34491)
0927 03:58:51.760113 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:51.760116 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:51.760261 (+ 145us) server_negotiation.cc:407] Connection header received
0927 03:58:51.760310 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:51.760311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:51.760321 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:51.760378 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:51.760801 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:51.761385 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:51.762502 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:51.762766 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:51.762858 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:51.763892 (+ 1034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:51.763894 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:51.763894 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:51.763901 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:51.765371 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:51.765740 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:51.765741 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:51.765742 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:51.765823 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:51.766024 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:51.766025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:51.766025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:51.766182 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:51.766262 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:51.766429 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 03:58:51.766508 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.185.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.185.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/krb5kdc/oryx_127.15.185.67.keytab.
Entry for principal oryx/127.15.185.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/krb5kdc/oryx_127.15.185.67.keytab.
I20260927 03:58:51.786957 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.67:0
--local_ip_for_outbound_sockets=127.15.185.67
--webserver_interface=127.15.185.67
--webserver_port=0
--tserver_master_addrs=127.15.185.126:34491
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/krb5kdc/oryx_127.15.185.67.keytab
--principal=oryx/127.15.185.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:42347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:58:51.869469 23974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:51.869697 23974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:51.869722 23974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:51.870887 23974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:51.870950 23974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:51.870971 23974 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 03:58:51.870991 23974 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.67
I20260927 03:58:51.872723 23974 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:42347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/krb5kdc/oryx_127.15.185.67.keytab
--principal=oryx/127.15.185.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.185.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.185.126:34491
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.67
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:51.873027 23974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:51.873279 23974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:51.878692 23979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:51.878720 23980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:51.878726 23982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:51.879158 23974 server_base.cc:1034] running on GCE node
I20260927 03:58:51.879379 23974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:51.879639 23974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:51.880797 23974 hybrid_clock.cc:648] HybridClock initialized: now 1790481531880775 us; error 29 us; skew 500 ppm
Sep 27 03:58:51 dist-test-slave-glgz krb5kdc[23577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481531, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.185.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:51.883459 23974 init.cc:377] Logged in from keytab as oryx/127.15.185.67@KRBTEST.COM (short username oryx)
I20260927 03:58:51.883865 23974 webserver.cc:460] Webserver started at http://127.15.185.67:39335/ using document root <none> and password file <none>
I20260927 03:58:51.884136 23974 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:51.884238 23974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:51.884351 23974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:51.885522 23974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/ts-2/data/instance:
uuid: "94088c272ea94c3f909bff38cd2bfbfb"
format_stamp: "Formatted at 2026-09-27 03:58:51 on dist-test-slave-glgz"
server_key: "f553113d46228c32a42d38dbd2842014"
server_key_iv: "3c2fc51715fb9709aaf92629f075a0f5"
server_key_version: "encryptionkey@0"
I20260927 03:58:51.885828 23974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/ts-2/wal/instance:
uuid: "94088c272ea94c3f909bff38cd2bfbfb"
format_stamp: "Formatted at 2026-09-27 03:58:51 on dist-test-slave-glgz"
server_key: "f553113d46228c32a42d38dbd2842014"
server_key_iv: "3c2fc51715fb9709aaf92629f075a0f5"
server_key_version: "encryptionkey@0"
I20260927 03:58:51.887197 23974 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 03:58:51.887959 23989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:51.888168 23974 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:58:51.888263 23974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/ts-2/wal
uuid: "94088c272ea94c3f909bff38cd2bfbfb"
format_stamp: "Formatted at 2026-09-27 03:58:51 on dist-test-slave-glgz"
server_key: "f553113d46228c32a42d38dbd2842014"
server_key_iv: "3c2fc51715fb9709aaf92629f075a0f5"
server_key_version: "encryptionkey@0"
I20260927 03:58:51.888341 23974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:51.905771 23974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:51.907788 23974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:51.907940 23974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:51.908476 23974 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:58:51.908546 23974 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:51.908581 23974 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:58:51.908604 23974 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:51.915304 23974 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.67:36625
I20260927 03:58:51.915388 24104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.67:36625 every 8 connection(s)
I20260927 03:58:51.915693 23974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
Sep 27 03:58:51 dist-test-slave-glgz krb5kdc[23577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481531, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.185.67@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM
I20260927 03:58:51.919181 23681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:51.909023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:56525 (local address 127.15.185.126:34491)
0927 03:58:51.909066 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:51.909068 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:51.909360 (+ 292us) server_negotiation.cc:407] Connection header received
0927 03:58:51.909826 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:51.909828 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:51.909842 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:51.909940 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:51.910851 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:51.911714 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:51.913342 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:51.913662 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:51.913742 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:51.916415 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:51.916416 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:51.916417 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:51.916423 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:51.917870 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:51.918282 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:51.918283 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:51.918283 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:51.918333 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:51.918559 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:51.918561 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:51.918561 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:51.918679 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:51.918746 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:51.918981 (+ 235us) server_negotiation.cc:299] Negotiation successful
0927 03:58:51.919106 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 03:58:51.919333 23998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:51.909033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34491 (local address 127.15.185.67:56525)
0927 03:58:51.909208 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:51.909245 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:51.909720 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:51.909999 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:51.910002 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:51.910433 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:51.910732 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:51.910739 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:51.911812 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:51.911813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:51.913247 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:51.913250 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:51.913759 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:51.913760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:51.913913 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:51.914615 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:51.914636 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:51.916321 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:51.918017 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:51.918022 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:51.918024 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:51.918222 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:51.918388 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:51.918389 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:51.918389 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:51.918478 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:51.918743 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:51.918747 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:51.918867 (+ 120us) client_negotiation.cc:769] Sending connection context
0927 03:58:51.918947 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 03:58:51.919126 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":85,"threads_started":1}
I20260927 03:58:51.919857 24105 heartbeater.cc:347] Connected to a master server at 127.15.185.126:34491
I20260927 03:58:51.919937 24105 heartbeater.cc:464] Registering TS with master...
I20260927 03:58:51.920141 24105 heartbeater.cc:511] Master 127.15.185.126:34491 requested a full tablet report, sending...
I20260927 03:58:51.920575 23621 ts_manager.cc:194] Registered new tserver with Master: 94088c272ea94c3f909bff38cd2bfbfb (127.15.185.67:36625)
I20260927 03:58:51.921092 23621 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.185.67@KRBTEST.COM'} at 127.15.185.67:56525
I20260927 03:58:51.922299 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 23974
I20260927 03:58:51.922413 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/ts-2/wal/instance
I20260927 03:58:51.922612 16101 external_mini_cluster.cc:1428] Setting key df793b176c08a6188e0712f1f8ae0a3e
I20260927 03:58:51.923614 16101 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:58:51.927408 23681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:51.920989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:42647 (local address 127.15.185.126:34491)
0927 03:58:51.921031 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:51.921033 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:51.921104 (+ 71us) server_negotiation.cc:407] Connection header received
0927 03:58:51.921177 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:51.921178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:51.921189 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:51.921243 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:51.921554 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:51.922008 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:51.923200 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:51.923457 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:51.923545 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:51.924595 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:51.924596 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:51.924596 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:51.924603 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:51.926144 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:51.926581 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:51.926581 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:51.926582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:51.926635 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:51.926833 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:51.926834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:51.926834 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:51.926977 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:51.927086 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:51.927249 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 03:58:51.927323 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 03:58:51.927460 24111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:51.920910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34491 (local address 127.15.185.67:42647)
0927 03:58:51.921047 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:51.921058 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:51.921117 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:51.921246 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:51.921247 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:51.921407 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:51.921480 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:51.921482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:51.922165 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:51.922166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:51.923126 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:51.923128 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:51.923548 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:51.923549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:51.923625 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:51.924177 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:51.924182 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:51.924486 (+ 304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:51.926232 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:51.926233 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:51.926234 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:51.926506 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:51.926701 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:51.926702 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:51.926703 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:51.926774 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:51.927076 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:51.927077 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:51.927127 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 03:58:51.927213 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 03:58:51.927322 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":89,"threads_started":1}
Sep 27 03:58:51 dist-test-slave-glgz krb5kdc[23577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481531, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 03:58:51 dist-test-slave-glgz krb5kdc[23577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:58:51 dist-test-slave-glgz krb5kdc[23577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
W20260927 03:58:51.945736 23681 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:58:51.940203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57220 (local address 127.15.185.126:34491)
0927 03:58:51.940270 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:51.940272 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:51.940282 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:58:51.940332 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:51.940333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:51.940349 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:51.940429 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:51.941047 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:51.941545 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:51.942325 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:51.942519 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:51.942591 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:51.945676 (+ 3085us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57220: BlockingRecv error: recv got EOF from 127.0.0.1:57220 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 27 03:58:51 dist-test-slave-glgz krb5kdc[23577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481531, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM
I20260927 03:58:51.954011 23681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:51.946943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57228 (local address 127.15.185.126:34491)
0927 03:58:51.947018 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:51.947020 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:51.947147 (+ 127us) server_negotiation.cc:407] Connection header received
0927 03:58:51.947229 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:51.947230 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:51.947240 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:51.947283 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:51.947776 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:51.948277 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:51.949058 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:51.949248 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:51.949332 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:51.951470 (+ 2138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:51.951471 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:51.951472 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:51.951480 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:51.952866 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:51.953226 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:51.953228 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:51.953228 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:51.953286 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:51.953494 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:51.953495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:51.953496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:51.953607 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:51.953708 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:51.953832 (+ 124us) server_negotiation.cc:299] Negotiation successful
0927 03:58:51.953908 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 03:58:51.955819 23621 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57228:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:58:51.957479 23621 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:58:51.964569 24130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:51.960180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37984 (local address 127.15.185.66:38731)
0927 03:58:51.960548 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:51.960554 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:51.960566 (+ 12us) server_negotiation.cc:407] Connection header received
0927 03:58:51.960608 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:51.960610 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:51.960627 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:51.960927 (+ 300us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:58:51.960988 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:51.961929 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:51.963628 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:51.964136 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:51.964237 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:51.964321 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:51.964421 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 03:58:51.964463 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":85,"threads_started":1}
I20260927 03:58:51.964622 24128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:51.960071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.65:46859 (local address 127.0.0.1:45574)
0927 03:58:51.960237 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:51.960270 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:51.960328 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:51.960679 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:51.960682 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:51.960684 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:58:51.960754 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:51.960756 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:51.962195 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:51.962196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:51.963384 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:51.963386 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:51.964357 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:51.964358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:51.964431 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:51.964435 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:58:51.964468 (+ 33us) client_negotiation.cc:240] Negotiation successful
0927 03:58:51.964508 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":38,"thread_start_us":88,"threads_started":1}
I20260927 03:58:51.964768 24129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:51.960055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.66:38731 (local address 127.0.0.1:37984)
0927 03:58:51.960217 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:51.960260 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:51.960325 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:51.960732 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:51.960733 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:51.960735 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:58:51.960813 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:51.960815 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:51.962028 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:51.962030 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:51.963569 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:51.963571 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:51.964240 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:51.964242 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:51.964320 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:51.964334 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 03:58:51.964408 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 03:58:51.964460 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":102,"thread_start_us":41,"threads_started":1}
I20260927 03:58:51.964840 24132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:51.960434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51824 (local address 127.15.185.67:36625)
0927 03:58:51.960626 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:51.960630 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:51.960659 (+ 29us) server_negotiation.cc:407] Connection header received
0927 03:58:51.960720 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:51.960723 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:51.960736 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:51.960811 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:58:51.960943 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:51.962054 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:51.963578 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:51.964281 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:51.964346 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:51.964422 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:51.964590 (+ 168us) server_negotiation.cc:299] Negotiation successful
0927 03:58:51.964718 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":92,"threads_started":1}
I20260927 03:58:51.965180 24127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:51.960103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.67:36625 (local address 127.0.0.1:51824)
0927 03:58:51.960601 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:51.960611 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:51.960648 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:51.960821 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:51.960822 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:51.960824 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:58:51.960862 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:51.960863 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:51.962143 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:51.962144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:51.963490 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:51.963492 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:51.964453 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:51.964454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:51.964506 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:51.964508 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 03:58:51.964560 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 03:58:51.964593 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"mutex_wait_us":81,"thread_start_us":64,"threads_started":1}
I20260927 03:58:51.965368 24131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:51.960314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45574 (local address 127.15.185.65:46859)
0927 03:58:51.960489 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:51.960495 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:51.960506 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:58:51.960546 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:51.960548 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:51.960563 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:51.960876 (+ 313us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:58:51.960937 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:51.962109 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:51.963470 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:51.964235 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:51.964922 (+ 687us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:51.965009 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:51.965188 (+ 179us) server_negotiation.cc:299] Negotiation successful
0927 03:58:51.965239 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":98,"threads_started":1}
I20260927 03:58:51.965613 23899 tablet_service.cc:1467] Processing CreateTablet for tablet 4f82ce3dd8e8427c9509da4385f3bf81 (DEFAULT_TABLE table=test-table [id=d0fb5ca494614fb78856404f17354aca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:58:51.965883 23759 tablet_service.cc:1467] Processing CreateTablet for tablet 4f82ce3dd8e8427c9509da4385f3bf81 (DEFAULT_TABLE table=test-table [id=d0fb5ca494614fb78856404f17354aca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:58:51.965991 23899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f82ce3dd8e8427c9509da4385f3bf81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:51.965862 24039 tablet_service.cc:1467] Processing CreateTablet for tablet 4f82ce3dd8e8427c9509da4385f3bf81 (DEFAULT_TABLE table=test-table [id=d0fb5ca494614fb78856404f17354aca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:58:51.966205 24039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f82ce3dd8e8427c9509da4385f3bf81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:51.966205 23759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f82ce3dd8e8427c9509da4385f3bf81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:51.969277 24133 tablet_bootstrap.cc:492] T 4f82ce3dd8e8427c9509da4385f3bf81 P 40c4cc7b32f14b0e89a6d4e89511a047: Bootstrap starting.
I20260927 03:58:51.969888 24134 tablet_bootstrap.cc:492] T 4f82ce3dd8e8427c9509da4385f3bf81 P 94088c272ea94c3f909bff38cd2bfbfb: Bootstrap starting.
I20260927 03:58:51.970023 24133 tablet_bootstrap.cc:654] T 4f82ce3dd8e8427c9509da4385f3bf81 P 40c4cc7b32f14b0e89a6d4e89511a047: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:51.970162 24135 tablet_bootstrap.cc:492] T 4f82ce3dd8e8427c9509da4385f3bf81 P 983a3b316b1f4809ae803c17f6c0def5: Bootstrap starting.
I20260927 03:58:51.970482 24133 log.cc:824] T 4f82ce3dd8e8427c9509da4385f3bf81 P 40c4cc7b32f14b0e89a6d4e89511a047: Log is configured to *not* fsync() on all Append() calls
I20260927 03:58:51.970536 24134 tablet_bootstrap.cc:654] T 4f82ce3dd8e8427c9509da4385f3bf81 P 94088c272ea94c3f909bff38cd2bfbfb: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:51.970770 24135 tablet_bootstrap.cc:654] T 4f82ce3dd8e8427c9509da4385f3bf81 P 983a3b316b1f4809ae803c17f6c0def5: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:51.970865 24134 log.cc:824] T 4f82ce3dd8e8427c9509da4385f3bf81 P 94088c272ea94c3f909bff38cd2bfbfb: Log is configured to *not* fsync() on all Append() calls
I20260927 03:58:51.971128 24135 log.cc:824] T 4f82ce3dd8e8427c9509da4385f3bf81 P 983a3b316b1f4809ae803c17f6c0def5: Log is configured to *not* fsync() on all Append() calls
I20260927 03:58:51.971338 24133 tablet_bootstrap.cc:492] T 4f82ce3dd8e8427c9509da4385f3bf81 P 40c4cc7b32f14b0e89a6d4e89511a047: No bootstrap required, opened a new log
I20260927 03:58:51.971411 24133 ts_tablet_manager.cc:1397] T 4f82ce3dd8e8427c9509da4385f3bf81 P 40c4cc7b32f14b0e89a6d4e89511a047: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:58:51.971747 24134 tablet_bootstrap.cc:492] T 4f82ce3dd8e8427c9509da4385f3bf81 P 94088c272ea94c3f909bff38cd2bfbfb: No bootstrap required, opened a new log
I20260927 03:58:51.971832 24134 ts_tablet_manager.cc:1397] T 4f82ce3dd8e8427c9509da4385f3bf81 P 94088c272ea94c3f909bff38cd2bfbfb: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 03:58:51.971820 24135 tablet_bootstrap.cc:492] T 4f82ce3dd8e8427c9509da4385f3bf81 P 983a3b316b1f4809ae803c17f6c0def5: No bootstrap required, opened a new log
I20260927 03:58:51.971895 24135 ts_tablet_manager.cc:1397] T 4f82ce3dd8e8427c9509da4385f3bf81 P 983a3b316b1f4809ae803c17f6c0def5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 03:58:51.973176 24134 raft_consensus.cc:348] T 4f82ce3dd8e8427c9509da4385f3bf81 P 94088c272ea94c3f909bff38cd2bfbfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983a3b316b1f4809ae803c17f6c0def5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46859 } } peers { permanent_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38731 } } peers { permanent_uuid: "94088c272ea94c3f909bff38cd2bfbfb" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36625 } }
I20260927 03:58:51.973335 24134 raft_consensus.cc:374] T 4f82ce3dd8e8427c9509da4385f3bf81 P 94088c272ea94c3f909bff38cd2bfbfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:51.973379 24134 raft_consensus.cc:729] T 4f82ce3dd8e8427c9509da4385f3bf81 P 94088c272ea94c3f909bff38cd2bfbfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94088c272ea94c3f909bff38cd2bfbfb, State: Initialized, Role: FOLLOWER
I20260927 03:58:51.973416 24133 raft_consensus.cc:348] T 4f82ce3dd8e8427c9509da4385f3bf81 P 40c4cc7b32f14b0e89a6d4e89511a047 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983a3b316b1f4809ae803c17f6c0def5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46859 } } peers { permanent_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38731 } } peers { permanent_uuid: "94088c272ea94c3f909bff38cd2bfbfb" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36625 } }
I20260927 03:58:51.973505 24134 consensus_queue.cc:260] T 4f82ce3dd8e8427c9509da4385f3bf81 P 94088c272ea94c3f909bff38cd2bfbfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983a3b316b1f4809ae803c17f6c0def5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46859 } } peers { permanent_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38731 } } peers { permanent_uuid: "94088c272ea94c3f909bff38cd2bfbfb" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36625 } }
I20260927 03:58:51.973594 24133 raft_consensus.cc:374] T 4f82ce3dd8e8427c9509da4385f3bf81 P 40c4cc7b32f14b0e89a6d4e89511a047 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:51.973628 24133 raft_consensus.cc:729] T 4f82ce3dd8e8427c9509da4385f3bf81 P 40c4cc7b32f14b0e89a6d4e89511a047 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c4cc7b32f14b0e89a6d4e89511a047, State: Initialized, Role: FOLLOWER
I20260927 03:58:51.973791 24134 ts_tablet_manager.cc:1428] T 4f82ce3dd8e8427c9509da4385f3bf81 P 94088c272ea94c3f909bff38cd2bfbfb: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 03:58:51.973788 24133 consensus_queue.cc:260] T 4f82ce3dd8e8427c9509da4385f3bf81 P 40c4cc7b32f14b0e89a6d4e89511a047 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983a3b316b1f4809ae803c17f6c0def5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46859 } } peers { permanent_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38731 } } peers { permanent_uuid: "94088c272ea94c3f909bff38cd2bfbfb" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36625 } }
I20260927 03:58:51.973968 24105 heartbeater.cc:503] Master 127.15.185.126:34491 was elected leader, sending a full tablet report...
I20260927 03:58:51.973865 24135 raft_consensus.cc:348] T 4f82ce3dd8e8427c9509da4385f3bf81 P 983a3b316b1f4809ae803c17f6c0def5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983a3b316b1f4809ae803c17f6c0def5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46859 } } peers { permanent_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38731 } } peers { permanent_uuid: "94088c272ea94c3f909bff38cd2bfbfb" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36625 } }
I20260927 03:58:51.974016 24135 raft_consensus.cc:374] T 4f82ce3dd8e8427c9509da4385f3bf81 P 983a3b316b1f4809ae803c17f6c0def5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:51.974057 24135 raft_consensus.cc:729] T 4f82ce3dd8e8427c9509da4385f3bf81 P 983a3b316b1f4809ae803c17f6c0def5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 983a3b316b1f4809ae803c17f6c0def5, State: Initialized, Role: FOLLOWER
I20260927 03:58:51.974073 24133 ts_tablet_manager.cc:1428] T 4f82ce3dd8e8427c9509da4385f3bf81 P 40c4cc7b32f14b0e89a6d4e89511a047: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:58:51.974184 23965 heartbeater.cc:503] Master 127.15.185.126:34491 was elected leader, sending a full tablet report...
I20260927 03:58:51.974174 24135 consensus_queue.cc:260] T 4f82ce3dd8e8427c9509da4385f3bf81 P 983a3b316b1f4809ae803c17f6c0def5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983a3b316b1f4809ae803c17f6c0def5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46859 } } peers { permanent_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38731 } } peers { permanent_uuid: "94088c272ea94c3f909bff38cd2bfbfb" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36625 } }
I20260927 03:58:51.974423 24135 ts_tablet_manager.cc:1428] T 4f82ce3dd8e8427c9509da4385f3bf81 P 983a3b316b1f4809ae803c17f6c0def5: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260927 03:58:51.974469 23825 heartbeater.cc:503] Master 127.15.185.126:34491 was elected leader, sending a full tablet report...
W20260927 03:58:52.005290 23966 tablet.cc:2369] T 4f82ce3dd8e8427c9509da4385f3bf81 P 40c4cc7b32f14b0e89a6d4e89511a047: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 03:58:52.086159 23826 tablet.cc:2369] T 4f82ce3dd8e8427c9509da4385f3bf81 P 983a3b316b1f4809ae803c17f6c0def5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:58:52.134351 24139 raft_consensus.cc:482] T 4f82ce3dd8e8427c9509da4385f3bf81 P 94088c272ea94c3f909bff38cd2bfbfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:58:52.134485 24139 raft_consensus.cc:504] T 4f82ce3dd8e8427c9509da4385f3bf81 P 94088c272ea94c3f909bff38cd2bfbfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983a3b316b1f4809ae803c17f6c0def5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46859 } } peers { permanent_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38731 } } peers { permanent_uuid: "94088c272ea94c3f909bff38cd2bfbfb" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36625 } }
I20260927 03:58:52.134838 24139 leader_election.cc:288] T 4f82ce3dd8e8427c9509da4385f3bf81 P 94088c272ea94c3f909bff38cd2bfbfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 983a3b316b1f4809ae803c17f6c0def5 (127.15.185.65:46859), 40c4cc7b32f14b0e89a6d4e89511a047 (127.15.185.66:38731)
I20260927 03:58:52.138950 24131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:52.135126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:53435 (local address 127.15.185.65:46859)
0927 03:58:52.135249 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:52.135251 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:52.135340 (+ 89us) server_negotiation.cc:407] Connection header received
0927 03:58:52.135369 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:52.135370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:52.135379 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:52.135430 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:58:52.135588 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:52.136334 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:52.137945 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:52.138536 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:52.138605 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:52.138615 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:52.138832 (+ 217us) server_negotiation.cc:299] Negotiation successful
0927 03:58:52.138872 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260927 03:58:52.138950 24130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:52.135139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:60913 (local address 127.15.185.66:38731)
0927 03:58:52.135201 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:52.135204 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:52.135216 (+ 12us) server_negotiation.cc:407] Connection header received
0927 03:58:52.135284 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:52.135285 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:52.135301 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:52.135392 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:58:52.135589 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:52.136334 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:52.137930 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:52.138522 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:52.138606 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:52.138615 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:52.138832 (+ 217us) server_negotiation.cc:299] Negotiation successful
0927 03:58:52.138871 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 03:58:52.138981 24142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:52.135076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.65:46859 (local address 127.15.185.67:53435)
0927 03:58:52.135264 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:52.135274 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:52.135331 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:52.135431 (+ 100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:52.135431 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:52.135432 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:58:52.135508 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:52.135510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:52.136437 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:52.136439 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:52.137810 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:52.137814 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:52.138640 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:52.138641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:52.138734 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:52.138739 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:58:52.138816 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 03:58:52.138871 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":60,"thread_start_us":109,"threads_started":1}
I20260927 03:58:52.139272 23998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:52.135049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.66:38731 (local address 127.15.185.67:60913)
0927 03:58:52.135144 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:52.135158 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:52.135215 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:52.135407 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:52.135408 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:52.135410 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:58:52.135508 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:52.135510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:52.136437 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:52.136439 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:52.137820 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:52.137821 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:52.138640 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:52.138641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:52.138737 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:52.138739 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 03:58:52.138816 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 03:58:52.138876 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73,"mutex_wait_us":118}
I20260927 03:58:52.139415 23779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f82ce3dd8e8427c9509da4385f3bf81" candidate_uuid: "94088c272ea94c3f909bff38cd2bfbfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "983a3b316b1f4809ae803c17f6c0def5" is_pre_election: true
I20260927 03:58:52.139607 23779 raft_consensus.cc:2454] T 4f82ce3dd8e8427c9509da4385f3bf81 P 983a3b316b1f4809ae803c17f6c0def5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94088c272ea94c3f909bff38cd2bfbfb in term 0.
I20260927 03:58:52.139631 23919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f82ce3dd8e8427c9509da4385f3bf81" candidate_uuid: "94088c272ea94c3f909bff38cd2bfbfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" is_pre_election: true
I20260927 03:58:52.139799 23919 raft_consensus.cc:2454] T 4f82ce3dd8e8427c9509da4385f3bf81 P 40c4cc7b32f14b0e89a6d4e89511a047 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94088c272ea94c3f909bff38cd2bfbfb in term 0.
I20260927 03:58:52.139835 23990 leader_election.cc:302] T 4f82ce3dd8e8427c9509da4385f3bf81 P 94088c272ea94c3f909bff38cd2bfbfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94088c272ea94c3f909bff38cd2bfbfb, 983a3b316b1f4809ae803c17f6c0def5; no voters:
I20260927 03:58:52.140007 24139 raft_consensus.cc:2781] T 4f82ce3dd8e8427c9509da4385f3bf81 P 94088c272ea94c3f909bff38cd2bfbfb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:58:52.140101 24139 raft_consensus.cc:482] T 4f82ce3dd8e8427c9509da4385f3bf81 P 94088c272ea94c3f909bff38cd2bfbfb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:58:52.140139 24139 raft_consensus.cc:3037] T 4f82ce3dd8e8427c9509da4385f3bf81 P 94088c272ea94c3f909bff38cd2bfbfb [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:52.141350 24139 raft_consensus.cc:504] T 4f82ce3dd8e8427c9509da4385f3bf81 P 94088c272ea94c3f909bff38cd2bfbfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983a3b316b1f4809ae803c17f6c0def5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46859 } } peers { permanent_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38731 } } peers { permanent_uuid: "94088c272ea94c3f909bff38cd2bfbfb" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36625 } }
I20260927 03:58:52.141526 24139 leader_election.cc:288] T 4f82ce3dd8e8427c9509da4385f3bf81 P 94088c272ea94c3f909bff38cd2bfbfb [CANDIDATE]: Term 1 election: Requested vote from peers 983a3b316b1f4809ae803c17f6c0def5 (127.15.185.65:46859), 40c4cc7b32f14b0e89a6d4e89511a047 (127.15.185.66:38731)
I20260927 03:58:52.141674 23779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f82ce3dd8e8427c9509da4385f3bf81" candidate_uuid: "94088c272ea94c3f909bff38cd2bfbfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "983a3b316b1f4809ae803c17f6c0def5"
I20260927 03:58:52.141700 23919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f82ce3dd8e8427c9509da4385f3bf81" candidate_uuid: "94088c272ea94c3f909bff38cd2bfbfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c4cc7b32f14b0e89a6d4e89511a047"
I20260927 03:58:52.141762 23779 raft_consensus.cc:3037] T 4f82ce3dd8e8427c9509da4385f3bf81 P 983a3b316b1f4809ae803c17f6c0def5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:52.141788 23919 raft_consensus.cc:3037] T 4f82ce3dd8e8427c9509da4385f3bf81 P 40c4cc7b32f14b0e89a6d4e89511a047 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:52.142472 23779 raft_consensus.cc:2454] T 4f82ce3dd8e8427c9509da4385f3bf81 P 983a3b316b1f4809ae803c17f6c0def5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94088c272ea94c3f909bff38cd2bfbfb in term 1.
I20260927 03:58:52.142483 23919 raft_consensus.cc:2454] T 4f82ce3dd8e8427c9509da4385f3bf81 P 40c4cc7b32f14b0e89a6d4e89511a047 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94088c272ea94c3f909bff38cd2bfbfb in term 1.
I20260927 03:58:52.142647 23990 leader_election.cc:302] T 4f82ce3dd8e8427c9509da4385f3bf81 P 94088c272ea94c3f909bff38cd2bfbfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94088c272ea94c3f909bff38cd2bfbfb, 983a3b316b1f4809ae803c17f6c0def5; no voters:
I20260927 03:58:52.142767 24139 raft_consensus.cc:2781] T 4f82ce3dd8e8427c9509da4385f3bf81 P 94088c272ea94c3f909bff38cd2bfbfb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:58:52.142936 24139 raft_consensus.cc:686] T 4f82ce3dd8e8427c9509da4385f3bf81 P 94088c272ea94c3f909bff38cd2bfbfb [term 1 LEADER]: Becoming Leader. State: Replica: 94088c272ea94c3f909bff38cd2bfbfb, State: Running, Role: LEADER
I20260927 03:58:52.143091 24139 consensus_queue.cc:237] T 4f82ce3dd8e8427c9509da4385f3bf81 P 94088c272ea94c3f909bff38cd2bfbfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983a3b316b1f4809ae803c17f6c0def5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46859 } } peers { permanent_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38731 } } peers { permanent_uuid: "94088c272ea94c3f909bff38cd2bfbfb" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36625 } }
I20260927 03:58:52.144025 23621 catalog_manager.cc:5495] T 4f82ce3dd8e8427c9509da4385f3bf81 P 94088c272ea94c3f909bff38cd2bfbfb reported cstate change: term changed from 0 to 1, leader changed from <none> to 94088c272ea94c3f909bff38cd2bfbfb (127.15.185.67). New cstate: current_term: 1 leader_uuid: "94088c272ea94c3f909bff38cd2bfbfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983a3b316b1f4809ae803c17f6c0def5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94088c272ea94c3f909bff38cd2bfbfb" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36625 } health_report { overall_health: HEALTHY } } }
W20260927 03:58:52.166455 24106 tablet.cc:2369] T 4f82ce3dd8e8427c9509da4385f3bf81 P 94088c272ea94c3f909bff38cd2bfbfb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:58:52.190651 24132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:52.187873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51828 (local address 127.15.185.67:36625)
0927 03:58:52.187941 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:52.187943 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:52.187954 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:58:52.188037 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:52.188038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:52.188060 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:52.188148 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:58:52.188309 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:52.188837 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:52.189689 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:52.190006 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:52.190100 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:52.190296 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:58:52.190415 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:58:52.190515 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:52.190537 (+ 22us) server_negotiation.cc:299] Negotiation successful
0927 03:58:52.190558 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 03:58:52.192498 23779 raft_consensus.cc:1261] T 4f82ce3dd8e8427c9509da4385f3bf81 P 983a3b316b1f4809ae803c17f6c0def5 [term 1 FOLLOWER]: Refusing update from remote peer 94088c272ea94c3f909bff38cd2bfbfb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:58:52.192480 23919 raft_consensus.cc:1261] T 4f82ce3dd8e8427c9509da4385f3bf81 P 40c4cc7b32f14b0e89a6d4e89511a047 [term 1 FOLLOWER]: Refusing update from remote peer 94088c272ea94c3f909bff38cd2bfbfb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:58:52.192828 24139 consensus_queue.cc:1035] T 4f82ce3dd8e8427c9509da4385f3bf81 P 94088c272ea94c3f909bff38cd2bfbfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:58:52.192994 24143 consensus_queue.cc:1035] T 4f82ce3dd8e8427c9509da4385f3bf81 P 94088c272ea94c3f909bff38cd2bfbfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "983a3b316b1f4809ae803c17f6c0def5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:58:52.195482 24147 mvcc.cc:205] Tried to move back new op lower bound from 7333812355857387520 to 7333812355658924032. Current Snapshot: MvccSnapshot[applied={T|T < 7333812355857387520}]
I20260927 03:58:52.197365 23621 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57228:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:58:52.197474 23621 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57228:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:58:52.198212 23621 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 523c89766816419d81500ea2272b97f5: Sending DeleteTablet for 3 replicas of tablet 4f82ce3dd8e8427c9509da4385f3bf81
I20260927 03:58:52.198547 24039 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f82ce3dd8e8427c9509da4385f3bf81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:58:52 UTC) from {username='oryx', principal='oryx/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:51824
I20260927 03:58:52.198585 23899 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f82ce3dd8e8427c9509da4385f3bf81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:58:52 UTC) from {username='oryx', principal='oryx/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:37984
I20260927 03:58:52.198604 23759 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f82ce3dd8e8427c9509da4385f3bf81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:58:52 UTC) from {username='oryx', principal='oryx/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:45574
I20260927 03:58:52.198757 24157 tablet_replica.cc:330] T 4f82ce3dd8e8427c9509da4385f3bf81 P 983a3b316b1f4809ae803c17f6c0def5: stopping tablet replica
I20260927 03:58:52.198781 24156 tablet_replica.cc:330] T 4f82ce3dd8e8427c9509da4385f3bf81 P 94088c272ea94c3f909bff38cd2bfbfb: stopping tablet replica
I20260927 03:58:52.198845 24156 raft_consensus.cc:2229] T 4f82ce3dd8e8427c9509da4385f3bf81 P 94088c272ea94c3f909bff38cd2bfbfb [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:58:52.198843 24157 raft_consensus.cc:2229] T 4f82ce3dd8e8427c9509da4385f3bf81 P 983a3b316b1f4809ae803c17f6c0def5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:58:52.198932 24156 raft_consensus.cc:2258] T 4f82ce3dd8e8427c9509da4385f3bf81 P 94088c272ea94c3f909bff38cd2bfbfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:58:52.198938 24157 raft_consensus.cc:2258] T 4f82ce3dd8e8427c9509da4385f3bf81 P 983a3b316b1f4809ae803c17f6c0def5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:58:52.199155 24158 tablet_replica.cc:330] T 4f82ce3dd8e8427c9509da4385f3bf81 P 40c4cc7b32f14b0e89a6d4e89511a047: stopping tablet replica
I20260927 03:58:52.199239 24158 raft_consensus.cc:2229] T 4f82ce3dd8e8427c9509da4385f3bf81 P 40c4cc7b32f14b0e89a6d4e89511a047 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:58:52.199380 23621 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57228:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:58:52.199544 23621 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:58:52.199715 24158 raft_consensus.cc:2258] T 4f82ce3dd8e8427c9509da4385f3bf81 P 40c4cc7b32f14b0e89a6d4e89511a047 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:58:52.199925 24156 ts_tablet_manager.cc:1905] T 4f82ce3dd8e8427c9509da4385f3bf81 P 94088c272ea94c3f909bff38cd2bfbfb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:58:52.200027 24157 ts_tablet_manager.cc:1905] T 4f82ce3dd8e8427c9509da4385f3bf81 P 983a3b316b1f4809ae803c17f6c0def5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:58:52.200994 23759 tablet_service.cc:1467] Processing CreateTablet for tablet 486f35c1d64e40459f1db6df0d1faff1 (DEFAULT_TABLE table=test-table [id=2bde981f3670455e8b976b758a0ed989]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:58:52.201119 23759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 486f35c1d64e40459f1db6df0d1faff1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:52.202008 24157 ts_tablet_manager.cc:1918] T 4f82ce3dd8e8427c9509da4385f3bf81 P 983a3b316b1f4809ae803c17f6c0def5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:58:52.202085 24157 log.cc:1196] T 4f82ce3dd8e8427c9509da4385f3bf81 P 983a3b316b1f4809ae803c17f6c0def5: Deleting WAL directory at /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/ts-0/wal/wals/4f82ce3dd8e8427c9509da4385f3bf81
I20260927 03:58:52.202047 24156 ts_tablet_manager.cc:1918] T 4f82ce3dd8e8427c9509da4385f3bf81 P 94088c272ea94c3f909bff38cd2bfbfb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:58:52.202265 24135 tablet_bootstrap.cc:492] T 486f35c1d64e40459f1db6df0d1faff1 P 983a3b316b1f4809ae803c17f6c0def5: Bootstrap starting.
I20260927 03:58:52.202348 24157 ts_tablet_manager.cc:1939] T 4f82ce3dd8e8427c9509da4385f3bf81 P 983a3b316b1f4809ae803c17f6c0def5: Deleting consensus metadata
I20260927 03:58:52.202342 24156 log.cc:1196] T 4f82ce3dd8e8427c9509da4385f3bf81 P 94088c272ea94c3f909bff38cd2bfbfb: Deleting WAL directory at /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/ts-2/wal/wals/4f82ce3dd8e8427c9509da4385f3bf81
I20260927 03:58:52.202646 24135 tablet_bootstrap.cc:654] T 486f35c1d64e40459f1db6df0d1faff1 P 983a3b316b1f4809ae803c17f6c0def5: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:52.202754 24156 ts_tablet_manager.cc:1939] T 4f82ce3dd8e8427c9509da4385f3bf81 P 94088c272ea94c3f909bff38cd2bfbfb: Deleting consensus metadata
I20260927 03:58:52.203248 23607 catalog_manager.cc:4841] TS 94088c272ea94c3f909bff38cd2bfbfb (127.15.185.67:36625): tablet 4f82ce3dd8e8427c9509da4385f3bf81 (table test-table [id=d0fb5ca494614fb78856404f17354aca]) successfully deleted
I20260927 03:58:52.203387 23606 catalog_manager.cc:4841] TS 983a3b316b1f4809ae803c17f6c0def5 (127.15.185.65:46859): tablet 4f82ce3dd8e8427c9509da4385f3bf81 (table test-table [id=d0fb5ca494614fb78856404f17354aca]) successfully deleted
I20260927 03:58:52.202191 24039 tablet_service.cc:1467] Processing CreateTablet for tablet 486f35c1d64e40459f1db6df0d1faff1 (DEFAULT_TABLE table=test-table [id=2bde981f3670455e8b976b758a0ed989]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:58:52.203593 24039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 486f35c1d64e40459f1db6df0d1faff1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:52.203630 23899 tablet_service.cc:1467] Processing CreateTablet for tablet 486f35c1d64e40459f1db6df0d1faff1 (DEFAULT_TABLE table=test-table [id=2bde981f3670455e8b976b758a0ed989]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:58:52.203720 23899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 486f35c1d64e40459f1db6df0d1faff1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:52.204811 24134 tablet_bootstrap.cc:492] T 486f35c1d64e40459f1db6df0d1faff1 P 94088c272ea94c3f909bff38cd2bfbfb: Bootstrap starting.
I20260927 03:58:52.205068 24133 tablet_bootstrap.cc:492] T 486f35c1d64e40459f1db6df0d1faff1 P 40c4cc7b32f14b0e89a6d4e89511a047: Bootstrap starting.
I20260927 03:58:52.205224 24134 tablet_bootstrap.cc:654] T 486f35c1d64e40459f1db6df0d1faff1 P 94088c272ea94c3f909bff38cd2bfbfb: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:52.205241 24135 tablet_bootstrap.cc:492] T 486f35c1d64e40459f1db6df0d1faff1 P 983a3b316b1f4809ae803c17f6c0def5: No bootstrap required, opened a new log
I20260927 03:58:52.205286 24135 ts_tablet_manager.cc:1397] T 486f35c1d64e40459f1db6df0d1faff1 P 983a3b316b1f4809ae803c17f6c0def5: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 03:58:52.205423 24135 raft_consensus.cc:348] T 486f35c1d64e40459f1db6df0d1faff1 P 983a3b316b1f4809ae803c17f6c0def5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983a3b316b1f4809ae803c17f6c0def5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46859 } } peers { permanent_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38731 } } peers { permanent_uuid: "94088c272ea94c3f909bff38cd2bfbfb" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36625 } }
I20260927 03:58:52.205482 24133 tablet_bootstrap.cc:654] T 486f35c1d64e40459f1db6df0d1faff1 P 40c4cc7b32f14b0e89a6d4e89511a047: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:52.205495 24135 raft_consensus.cc:374] T 486f35c1d64e40459f1db6df0d1faff1 P 983a3b316b1f4809ae803c17f6c0def5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:52.205518 24135 raft_consensus.cc:729] T 486f35c1d64e40459f1db6df0d1faff1 P 983a3b316b1f4809ae803c17f6c0def5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 983a3b316b1f4809ae803c17f6c0def5, State: Initialized, Role: FOLLOWER
I20260927 03:58:52.205564 24135 consensus_queue.cc:260] T 486f35c1d64e40459f1db6df0d1faff1 P 983a3b316b1f4809ae803c17f6c0def5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983a3b316b1f4809ae803c17f6c0def5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46859 } } peers { permanent_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38731 } } peers { permanent_uuid: "94088c272ea94c3f909bff38cd2bfbfb" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36625 } }
I20260927 03:58:52.205669 24135 ts_tablet_manager.cc:1428] T 486f35c1d64e40459f1db6df0d1faff1 P 983a3b316b1f4809ae803c17f6c0def5: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 03:58:52.206161 24133 tablet_bootstrap.cc:492] T 486f35c1d64e40459f1db6df0d1faff1 P 40c4cc7b32f14b0e89a6d4e89511a047: No bootstrap required, opened a new log
I20260927 03:58:52.206213 24133 ts_tablet_manager.cc:1397] T 486f35c1d64e40459f1db6df0d1faff1 P 40c4cc7b32f14b0e89a6d4e89511a047: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:58:52.206351 24133 raft_consensus.cc:348] T 486f35c1d64e40459f1db6df0d1faff1 P 40c4cc7b32f14b0e89a6d4e89511a047 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983a3b316b1f4809ae803c17f6c0def5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46859 } } peers { permanent_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38731 } } peers { permanent_uuid: "94088c272ea94c3f909bff38cd2bfbfb" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36625 } }
I20260927 03:58:52.206426 24133 raft_consensus.cc:374] T 486f35c1d64e40459f1db6df0d1faff1 P 40c4cc7b32f14b0e89a6d4e89511a047 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:52.206430 24134 tablet_bootstrap.cc:492] T 486f35c1d64e40459f1db6df0d1faff1 P 94088c272ea94c3f909bff38cd2bfbfb: No bootstrap required, opened a new log
I20260927 03:58:52.206449 24133 raft_consensus.cc:729] T 486f35c1d64e40459f1db6df0d1faff1 P 40c4cc7b32f14b0e89a6d4e89511a047 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c4cc7b32f14b0e89a6d4e89511a047, State: Initialized, Role: FOLLOWER
I20260927 03:58:52.206475 24134 ts_tablet_manager.cc:1397] T 486f35c1d64e40459f1db6df0d1faff1 P 94088c272ea94c3f909bff38cd2bfbfb: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 03:58:52.206501 24133 consensus_queue.cc:260] T 486f35c1d64e40459f1db6df0d1faff1 P 40c4cc7b32f14b0e89a6d4e89511a047 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983a3b316b1f4809ae803c17f6c0def5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46859 } } peers { permanent_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38731 } } peers { permanent_uuid: "94088c272ea94c3f909bff38cd2bfbfb" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36625 } }
I20260927 03:58:52.206588 24133 ts_tablet_manager.cc:1428] T 486f35c1d64e40459f1db6df0d1faff1 P 40c4cc7b32f14b0e89a6d4e89511a047: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:52.206629 24134 raft_consensus.cc:348] T 486f35c1d64e40459f1db6df0d1faff1 P 94088c272ea94c3f909bff38cd2bfbfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983a3b316b1f4809ae803c17f6c0def5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46859 } } peers { permanent_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38731 } } peers { permanent_uuid: "94088c272ea94c3f909bff38cd2bfbfb" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36625 } }
I20260927 03:58:52.206707 24134 raft_consensus.cc:374] T 486f35c1d64e40459f1db6df0d1faff1 P 94088c272ea94c3f909bff38cd2bfbfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:52.206730 24134 raft_consensus.cc:729] T 486f35c1d64e40459f1db6df0d1faff1 P 94088c272ea94c3f909bff38cd2bfbfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94088c272ea94c3f909bff38cd2bfbfb, State: Initialized, Role: FOLLOWER
I20260927 03:58:52.206789 24134 consensus_queue.cc:260] T 486f35c1d64e40459f1db6df0d1faff1 P 94088c272ea94c3f909bff38cd2bfbfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983a3b316b1f4809ae803c17f6c0def5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46859 } } peers { permanent_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38731 } } peers { permanent_uuid: "94088c272ea94c3f909bff38cd2bfbfb" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36625 } }
I20260927 03:58:52.206873 24134 ts_tablet_manager.cc:1428] T 486f35c1d64e40459f1db6df0d1faff1 P 94088c272ea94c3f909bff38cd2bfbfb: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260927 03:58:52.207258 24158 ts_tablet_manager.cc:1905] T 4f82ce3dd8e8427c9509da4385f3bf81 P 40c4cc7b32f14b0e89a6d4e89511a047: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:58:52.208647 24158 ts_tablet_manager.cc:1918] T 4f82ce3dd8e8427c9509da4385f3bf81 P 40c4cc7b32f14b0e89a6d4e89511a047: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:58:52.208714 24158 log.cc:1196] T 4f82ce3dd8e8427c9509da4385f3bf81 P 40c4cc7b32f14b0e89a6d4e89511a047: Deleting WAL directory at /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/ts-1/wal/wals/4f82ce3dd8e8427c9509da4385f3bf81
I20260927 03:58:52.208906 24158 ts_tablet_manager.cc:1939] T 4f82ce3dd8e8427c9509da4385f3bf81 P 40c4cc7b32f14b0e89a6d4e89511a047: Deleting consensus metadata
I20260927 03:58:52.209250 23609 catalog_manager.cc:4841] TS 40c4cc7b32f14b0e89a6d4e89511a047 (127.15.185.66:38731): tablet 4f82ce3dd8e8427c9509da4385f3bf81 (table test-table [id=d0fb5ca494614fb78856404f17354aca]) successfully deleted
I20260927 03:58:52.265631 24141 raft_consensus.cc:482] T 486f35c1d64e40459f1db6df0d1faff1 P 983a3b316b1f4809ae803c17f6c0def5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:58:52.265748 24141 raft_consensus.cc:504] T 486f35c1d64e40459f1db6df0d1faff1 P 983a3b316b1f4809ae803c17f6c0def5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983a3b316b1f4809ae803c17f6c0def5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46859 } } peers { permanent_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38731 } } peers { permanent_uuid: "94088c272ea94c3f909bff38cd2bfbfb" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36625 } }
I20260927 03:58:52.266105 24141 leader_election.cc:288] T 486f35c1d64e40459f1db6df0d1faff1 P 983a3b316b1f4809ae803c17f6c0def5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40c4cc7b32f14b0e89a6d4e89511a047 (127.15.185.66:38731), 94088c272ea94c3f909bff38cd2bfbfb (127.15.185.67:36625)
I20260927 03:58:52.269905 24132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:52.266499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:34905 (local address 127.15.185.67:36625)
0927 03:58:52.266546 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:52.266549 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:52.266558 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:58:52.266586 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:52.266587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:52.266596 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:52.266660 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:58:52.266849 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:52.267611 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:52.269124 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:52.269479 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:52.269572 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:52.269584 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:52.269789 (+ 205us) server_negotiation.cc:299] Negotiation successful
0927 03:58:52.269833 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 03:58:52.270026 24130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:52.266322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:58341 (local address 127.15.185.66:38731)
0927 03:58:52.266368 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:52.266370 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:52.266491 (+ 121us) server_negotiation.cc:407] Connection header received
0927 03:58:52.266589 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:52.266590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:52.266598 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:52.266648 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:58:52.266849 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:52.267610 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:52.269102 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:52.269649 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:52.269730 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:52.269739 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:52.269894 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 03:58:52.269943 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 03:58:52.269937 24163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:52.266296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.67:36625 (local address 127.15.185.65:34905)
0927 03:58:52.266453 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:52.266462 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:52.266506 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:52.266672 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:52.266674 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:52.266676 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:58:52.266758 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:52.266760 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:52.267716 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:52.267716 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:52.268998 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:52.269001 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:52.269590 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:52.269592 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:52.269672 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:52.269675 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:58:52.269774 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 03:58:52.269823 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":23,"thread_start_us":105,"threads_started":1}
I20260927 03:58:52.270272 24162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:52.266289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.66:38731 (local address 127.15.185.65:58341)
0927 03:58:52.266419 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:52.266432 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:52.266511 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:52.266713 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:52.266713 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:52.266715 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:58:52.266768 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:52.266769 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:52.267716 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:52.267716 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:52.268998 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:52.269001 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:52.269739 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:52.269740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:52.269806 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:52.269809 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:58:52.270143 (+ 334us) client_negotiation.cc:240] Negotiation successful
0927 03:58:52.270191 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":82,"thread_start_us":63,"threads_started":1}
I20260927 03:58:52.270541 23919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "486f35c1d64e40459f1db6df0d1faff1" candidate_uuid: "983a3b316b1f4809ae803c17f6c0def5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" is_pre_election: true
I20260927 03:58:52.270576 24059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "486f35c1d64e40459f1db6df0d1faff1" candidate_uuid: "983a3b316b1f4809ae803c17f6c0def5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94088c272ea94c3f909bff38cd2bfbfb" is_pre_election: true
I20260927 03:58:52.270648 23919 raft_consensus.cc:2454] T 486f35c1d64e40459f1db6df0d1faff1 P 40c4cc7b32f14b0e89a6d4e89511a047 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 983a3b316b1f4809ae803c17f6c0def5 in term 0.
I20260927 03:58:52.270680 24059 raft_consensus.cc:2454] T 486f35c1d64e40459f1db6df0d1faff1 P 94088c272ea94c3f909bff38cd2bfbfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 983a3b316b1f4809ae803c17f6c0def5 in term 0.
I20260927 03:58:52.270818 23713 leader_election.cc:302] T 486f35c1d64e40459f1db6df0d1faff1 P 983a3b316b1f4809ae803c17f6c0def5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c4cc7b32f14b0e89a6d4e89511a047, 983a3b316b1f4809ae803c17f6c0def5; no voters:
I20260927 03:58:52.270973 24141 raft_consensus.cc:2781] T 486f35c1d64e40459f1db6df0d1faff1 P 983a3b316b1f4809ae803c17f6c0def5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:58:52.271020 24141 raft_consensus.cc:482] T 486f35c1d64e40459f1db6df0d1faff1 P 983a3b316b1f4809ae803c17f6c0def5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:58:52.271035 24141 raft_consensus.cc:3037] T 486f35c1d64e40459f1db6df0d1faff1 P 983a3b316b1f4809ae803c17f6c0def5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:52.271764 24141 raft_consensus.cc:504] T 486f35c1d64e40459f1db6df0d1faff1 P 983a3b316b1f4809ae803c17f6c0def5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983a3b316b1f4809ae803c17f6c0def5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46859 } } peers { permanent_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38731 } } peers { permanent_uuid: "94088c272ea94c3f909bff38cd2bfbfb" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36625 } }
I20260927 03:58:52.271907 24141 leader_election.cc:288] T 486f35c1d64e40459f1db6df0d1faff1 P 983a3b316b1f4809ae803c17f6c0def5 [CANDIDATE]: Term 1 election: Requested vote from peers 40c4cc7b32f14b0e89a6d4e89511a047 (127.15.185.66:38731), 94088c272ea94c3f909bff38cd2bfbfb (127.15.185.67:36625)
I20260927 03:58:52.272042 23919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "486f35c1d64e40459f1db6df0d1faff1" candidate_uuid: "983a3b316b1f4809ae803c17f6c0def5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c4cc7b32f14b0e89a6d4e89511a047"
I20260927 03:58:52.272117 23919 raft_consensus.cc:3037] T 486f35c1d64e40459f1db6df0d1faff1 P 40c4cc7b32f14b0e89a6d4e89511a047 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:52.272096 24059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "486f35c1d64e40459f1db6df0d1faff1" candidate_uuid: "983a3b316b1f4809ae803c17f6c0def5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94088c272ea94c3f909bff38cd2bfbfb"
I20260927 03:58:52.272176 24059 raft_consensus.cc:3037] T 486f35c1d64e40459f1db6df0d1faff1 P 94088c272ea94c3f909bff38cd2bfbfb [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:52.272688 23919 raft_consensus.cc:2454] T 486f35c1d64e40459f1db6df0d1faff1 P 40c4cc7b32f14b0e89a6d4e89511a047 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 983a3b316b1f4809ae803c17f6c0def5 in term 1.
I20260927 03:58:52.272852 23713 leader_election.cc:302] T 486f35c1d64e40459f1db6df0d1faff1 P 983a3b316b1f4809ae803c17f6c0def5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c4cc7b32f14b0e89a6d4e89511a047, 983a3b316b1f4809ae803c17f6c0def5; no voters:
I20260927 03:58:52.272933 24141 raft_consensus.cc:2781] T 486f35c1d64e40459f1db6df0d1faff1 P 983a3b316b1f4809ae803c17f6c0def5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:58:52.273000 24059 raft_consensus.cc:2454] T 486f35c1d64e40459f1db6df0d1faff1 P 94088c272ea94c3f909bff38cd2bfbfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 983a3b316b1f4809ae803c17f6c0def5 in term 1.
I20260927 03:58:52.273063 24141 raft_consensus.cc:686] T 486f35c1d64e40459f1db6df0d1faff1 P 983a3b316b1f4809ae803c17f6c0def5 [term 1 LEADER]: Becoming Leader. State: Replica: 983a3b316b1f4809ae803c17f6c0def5, State: Running, Role: LEADER
I20260927 03:58:52.273131 24141 consensus_queue.cc:237] T 486f35c1d64e40459f1db6df0d1faff1 P 983a3b316b1f4809ae803c17f6c0def5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983a3b316b1f4809ae803c17f6c0def5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46859 } } peers { permanent_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38731 } } peers { permanent_uuid: "94088c272ea94c3f909bff38cd2bfbfb" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36625 } }
I20260927 03:58:52.273916 23621 catalog_manager.cc:5495] T 486f35c1d64e40459f1db6df0d1faff1 P 983a3b316b1f4809ae803c17f6c0def5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 983a3b316b1f4809ae803c17f6c0def5 (127.15.185.65). New cstate: current_term: 1 leader_uuid: "983a3b316b1f4809ae803c17f6c0def5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983a3b316b1f4809ae803c17f6c0def5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94088c272ea94c3f909bff38cd2bfbfb" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36625 } health_report { overall_health: UNKNOWN } } }
Sep 27 03:58:52 dist-test-slave-glgz krb5kdc[23577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481531, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.185.126@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM
I20260927 03:58:52.311539 24171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:52.303837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34491 (local address 127.0.0.1:57238)
0927 03:58:52.304056 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:52.304066 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:52.304128 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:52.304284 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:52.304284 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:52.304583 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:52.304631 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:52.304633 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:52.305568 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:52.305569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:52.306104 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:52.306105 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:52.306604 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:52.306605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:52.306646 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:52.307273 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:52.307278 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:52.308857 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:52.310529 (+ 1672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:52.310536 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:52.310537 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:52.310726 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:52.310922 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:52.310923 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:52.310923 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:52.310977 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:52.311258 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:52.311264 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:52.311309 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 03:58:52.311365 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 03:58:52.311460 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":96,"threads_started":1}
I20260927 03:58:52.311637 23681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:52.303966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57238 (local address 127.15.185.126:34491)
0927 03:58:52.304049 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:52.304051 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:52.304140 (+ 89us) server_negotiation.cc:407] Connection header received
0927 03:58:52.304173 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:52.304174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:52.304186 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:52.304270 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:52.304729 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:52.305473 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:52.306221 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:52.306522 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:52.306601 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:52.308942 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:52.308943 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:52.308944 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:52.308951 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:52.310445 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:52.310798 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:52.310799 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:52.310800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:52.310859 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:52.311069 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:52.311070 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:52.311070 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:52.311198 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:52.311261 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:52.311415 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 03:58:52.311522 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":53}
I20260927 03:58:52.313899 23621 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:57238:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 03:58:52.314127 23621 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:58:52.316221 23899 tablet_service.cc:1467] Processing CreateTablet for tablet 89bd7cdb9917467ebc03e7890a98794d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b7a286dae45341bab7f0553a750c930c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 03:58:52.316409 23899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89bd7cdb9917467ebc03e7890a98794d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:52.316425 24039 tablet_service.cc:1467] Processing CreateTablet for tablet 89bd7cdb9917467ebc03e7890a98794d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b7a286dae45341bab7f0553a750c930c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 03:58:52.316910 24039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89bd7cdb9917467ebc03e7890a98794d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:52.316888 23759 tablet_service.cc:1467] Processing CreateTablet for tablet 89bd7cdb9917467ebc03e7890a98794d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b7a286dae45341bab7f0553a750c930c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 03:58:52.317034 23759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89bd7cdb9917467ebc03e7890a98794d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:52.318049 24133 tablet_bootstrap.cc:492] T 89bd7cdb9917467ebc03e7890a98794d P 40c4cc7b32f14b0e89a6d4e89511a047: Bootstrap starting.
I20260927 03:58:52.318405 24134 tablet_bootstrap.cc:492] T 89bd7cdb9917467ebc03e7890a98794d P 94088c272ea94c3f909bff38cd2bfbfb: Bootstrap starting.
I20260927 03:58:52.318572 24133 tablet_bootstrap.cc:654] T 89bd7cdb9917467ebc03e7890a98794d P 40c4cc7b32f14b0e89a6d4e89511a047: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:52.318686 24134 tablet_bootstrap.cc:654] T 89bd7cdb9917467ebc03e7890a98794d P 94088c272ea94c3f909bff38cd2bfbfb: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:52.318951 24135 tablet_bootstrap.cc:492] T 89bd7cdb9917467ebc03e7890a98794d P 983a3b316b1f4809ae803c17f6c0def5: Bootstrap starting.
I20260927 03:58:52.319271 24133 tablet_bootstrap.cc:492] T 89bd7cdb9917467ebc03e7890a98794d P 40c4cc7b32f14b0e89a6d4e89511a047: No bootstrap required, opened a new log
I20260927 03:58:52.319315 24133 ts_tablet_manager.cc:1397] T 89bd7cdb9917467ebc03e7890a98794d P 40c4cc7b32f14b0e89a6d4e89511a047: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 03:58:52.319410 24135 tablet_bootstrap.cc:654] T 89bd7cdb9917467ebc03e7890a98794d P 983a3b316b1f4809ae803c17f6c0def5: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:52.319480 24133 raft_consensus.cc:348] T 89bd7cdb9917467ebc03e7890a98794d P 40c4cc7b32f14b0e89a6d4e89511a047 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983a3b316b1f4809ae803c17f6c0def5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46859 } } peers { permanent_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38731 } } peers { permanent_uuid: "94088c272ea94c3f909bff38cd2bfbfb" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36625 } }
I20260927 03:58:52.319566 24133 raft_consensus.cc:374] T 89bd7cdb9917467ebc03e7890a98794d P 40c4cc7b32f14b0e89a6d4e89511a047 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:52.319590 24133 raft_consensus.cc:729] T 89bd7cdb9917467ebc03e7890a98794d P 40c4cc7b32f14b0e89a6d4e89511a047 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c4cc7b32f14b0e89a6d4e89511a047, State: Initialized, Role: FOLLOWER
I20260927 03:58:52.319648 24133 consensus_queue.cc:260] T 89bd7cdb9917467ebc03e7890a98794d P 40c4cc7b32f14b0e89a6d4e89511a047 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983a3b316b1f4809ae803c17f6c0def5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46859 } } peers { permanent_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38731 } } peers { permanent_uuid: "94088c272ea94c3f909bff38cd2bfbfb" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36625 } }
I20260927 03:58:52.319767 24133 ts_tablet_manager.cc:1428] T 89bd7cdb9917467ebc03e7890a98794d P 40c4cc7b32f14b0e89a6d4e89511a047: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:52.319839 24140 tablet_replica.cc:439] T 89bd7cdb9917467ebc03e7890a98794d P 40c4cc7b32f14b0e89a6d4e89511a047: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983a3b316b1f4809ae803c17f6c0def5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46859 } } peers { permanent_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38731 } } peers { permanent_uuid: "94088c272ea94c3f909bff38cd2bfbfb" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36625 } } }
I20260927 03:58:52.319944 24140 tablet_replica.cc:442] T 89bd7cdb9917467ebc03e7890a98794d P 40c4cc7b32f14b0e89a6d4e89511a047: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:58:52.320602 24134 tablet_bootstrap.cc:492] T 89bd7cdb9917467ebc03e7890a98794d P 94088c272ea94c3f909bff38cd2bfbfb: No bootstrap required, opened a new log
I20260927 03:58:52.320654 24134 ts_tablet_manager.cc:1397] T 89bd7cdb9917467ebc03e7890a98794d P 94088c272ea94c3f909bff38cd2bfbfb: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 03:58:52.320757 24134 raft_consensus.cc:348] T 89bd7cdb9917467ebc03e7890a98794d P 94088c272ea94c3f909bff38cd2bfbfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983a3b316b1f4809ae803c17f6c0def5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46859 } } peers { permanent_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38731 } } peers { permanent_uuid: "94088c272ea94c3f909bff38cd2bfbfb" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36625 } }
I20260927 03:58:52.320848 24134 raft_consensus.cc:374] T 89bd7cdb9917467ebc03e7890a98794d P 94088c272ea94c3f909bff38cd2bfbfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:52.320869 24134 raft_consensus.cc:729] T 89bd7cdb9917467ebc03e7890a98794d P 94088c272ea94c3f909bff38cd2bfbfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94088c272ea94c3f909bff38cd2bfbfb, State: Initialized, Role: FOLLOWER
I20260927 03:58:52.320921 24134 consensus_queue.cc:260] T 89bd7cdb9917467ebc03e7890a98794d P 94088c272ea94c3f909bff38cd2bfbfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983a3b316b1f4809ae803c17f6c0def5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46859 } } peers { permanent_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38731 } } peers { permanent_uuid: "94088c272ea94c3f909bff38cd2bfbfb" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36625 } }
I20260927 03:58:52.321033 24134 ts_tablet_manager.cc:1428] T 89bd7cdb9917467ebc03e7890a98794d P 94088c272ea94c3f909bff38cd2bfbfb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:52.321049 24151 tablet_replica.cc:439] T 89bd7cdb9917467ebc03e7890a98794d P 94088c272ea94c3f909bff38cd2bfbfb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983a3b316b1f4809ae803c17f6c0def5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46859 } } peers { permanent_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38731 } } peers { permanent_uuid: "94088c272ea94c3f909bff38cd2bfbfb" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36625 } } }
I20260927 03:58:52.321120 24151 tablet_replica.cc:442] T 89bd7cdb9917467ebc03e7890a98794d P 94088c272ea94c3f909bff38cd2bfbfb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:58:52.322049 24135 tablet_bootstrap.cc:492] T 89bd7cdb9917467ebc03e7890a98794d P 983a3b316b1f4809ae803c17f6c0def5: No bootstrap required, opened a new log
I20260927 03:58:52.322115 24135 ts_tablet_manager.cc:1397] T 89bd7cdb9917467ebc03e7890a98794d P 983a3b316b1f4809ae803c17f6c0def5: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 03:58:52.322264 24135 raft_consensus.cc:348] T 89bd7cdb9917467ebc03e7890a98794d P 983a3b316b1f4809ae803c17f6c0def5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983a3b316b1f4809ae803c17f6c0def5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46859 } } peers { permanent_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38731 } } peers { permanent_uuid: "94088c272ea94c3f909bff38cd2bfbfb" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36625 } }
I20260927 03:58:52.322364 24135 raft_consensus.cc:374] T 89bd7cdb9917467ebc03e7890a98794d P 983a3b316b1f4809ae803c17f6c0def5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:52.322397 24135 raft_consensus.cc:729] T 89bd7cdb9917467ebc03e7890a98794d P 983a3b316b1f4809ae803c17f6c0def5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 983a3b316b1f4809ae803c17f6c0def5, State: Initialized, Role: FOLLOWER
I20260927 03:58:52.322453 24135 consensus_queue.cc:260] T 89bd7cdb9917467ebc03e7890a98794d P 983a3b316b1f4809ae803c17f6c0def5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983a3b316b1f4809ae803c17f6c0def5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46859 } } peers { permanent_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38731 } } peers { permanent_uuid: "94088c272ea94c3f909bff38cd2bfbfb" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36625 } }
I20260927 03:58:52.322556 24135 ts_tablet_manager.cc:1428] T 89bd7cdb9917467ebc03e7890a98794d P 983a3b316b1f4809ae803c17f6c0def5: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 03:58:52.322692 24164 tablet_replica.cc:439] T 89bd7cdb9917467ebc03e7890a98794d P 983a3b316b1f4809ae803c17f6c0def5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983a3b316b1f4809ae803c17f6c0def5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46859 } } peers { permanent_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38731 } } peers { permanent_uuid: "94088c272ea94c3f909bff38cd2bfbfb" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36625 } } }
I20260927 03:58:52.322817 24164 tablet_replica.cc:442] T 89bd7cdb9917467ebc03e7890a98794d P 983a3b316b1f4809ae803c17f6c0def5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:58:52.376915 24164 raft_consensus.cc:482] T 89bd7cdb9917467ebc03e7890a98794d P 983a3b316b1f4809ae803c17f6c0def5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:58:52.376998 24164 raft_consensus.cc:504] T 89bd7cdb9917467ebc03e7890a98794d P 983a3b316b1f4809ae803c17f6c0def5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983a3b316b1f4809ae803c17f6c0def5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46859 } } peers { permanent_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38731 } } peers { permanent_uuid: "94088c272ea94c3f909bff38cd2bfbfb" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36625 } }
I20260927 03:58:52.377185 24164 leader_election.cc:288] T 89bd7cdb9917467ebc03e7890a98794d P 983a3b316b1f4809ae803c17f6c0def5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40c4cc7b32f14b0e89a6d4e89511a047 (127.15.185.66:38731), 94088c272ea94c3f909bff38cd2bfbfb (127.15.185.67:36625)
I20260927 03:58:52.377436 23919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89bd7cdb9917467ebc03e7890a98794d" candidate_uuid: "983a3b316b1f4809ae803c17f6c0def5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" is_pre_election: true
I20260927 03:58:52.377543 23919 raft_consensus.cc:2454] T 89bd7cdb9917467ebc03e7890a98794d P 40c4cc7b32f14b0e89a6d4e89511a047 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 983a3b316b1f4809ae803c17f6c0def5 in term 0.
I20260927 03:58:52.377471 24059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89bd7cdb9917467ebc03e7890a98794d" candidate_uuid: "983a3b316b1f4809ae803c17f6c0def5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94088c272ea94c3f909bff38cd2bfbfb" is_pre_election: true
I20260927 03:58:52.377591 24059 raft_consensus.cc:2454] T 89bd7cdb9917467ebc03e7890a98794d P 94088c272ea94c3f909bff38cd2bfbfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 983a3b316b1f4809ae803c17f6c0def5 in term 0.
I20260927 03:58:52.377718 23713 leader_election.cc:302] T 89bd7cdb9917467ebc03e7890a98794d P 983a3b316b1f4809ae803c17f6c0def5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c4cc7b32f14b0e89a6d4e89511a047, 983a3b316b1f4809ae803c17f6c0def5; no voters:
I20260927 03:58:52.377821 24164 raft_consensus.cc:2781] T 89bd7cdb9917467ebc03e7890a98794d P 983a3b316b1f4809ae803c17f6c0def5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:58:52.377872 24164 raft_consensus.cc:482] T 89bd7cdb9917467ebc03e7890a98794d P 983a3b316b1f4809ae803c17f6c0def5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:58:52.377905 24164 raft_consensus.cc:3037] T 89bd7cdb9917467ebc03e7890a98794d P 983a3b316b1f4809ae803c17f6c0def5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:52.378587 24164 raft_consensus.cc:504] T 89bd7cdb9917467ebc03e7890a98794d P 983a3b316b1f4809ae803c17f6c0def5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983a3b316b1f4809ae803c17f6c0def5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46859 } } peers { permanent_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38731 } } peers { permanent_uuid: "94088c272ea94c3f909bff38cd2bfbfb" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36625 } }
I20260927 03:58:52.378726 24164 leader_election.cc:288] T 89bd7cdb9917467ebc03e7890a98794d P 983a3b316b1f4809ae803c17f6c0def5 [CANDIDATE]: Term 1 election: Requested vote from peers 40c4cc7b32f14b0e89a6d4e89511a047 (127.15.185.66:38731), 94088c272ea94c3f909bff38cd2bfbfb (127.15.185.67:36625)
I20260927 03:58:52.378868 23919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89bd7cdb9917467ebc03e7890a98794d" candidate_uuid: "983a3b316b1f4809ae803c17f6c0def5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c4cc7b32f14b0e89a6d4e89511a047"
I20260927 03:58:52.378940 23919 raft_consensus.cc:3037] T 89bd7cdb9917467ebc03e7890a98794d P 40c4cc7b32f14b0e89a6d4e89511a047 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:52.378875 24059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89bd7cdb9917467ebc03e7890a98794d" candidate_uuid: "983a3b316b1f4809ae803c17f6c0def5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94088c272ea94c3f909bff38cd2bfbfb"
I20260927 03:58:52.379025 24059 raft_consensus.cc:3037] T 89bd7cdb9917467ebc03e7890a98794d P 94088c272ea94c3f909bff38cd2bfbfb [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:52.379798 23919 raft_consensus.cc:2454] T 89bd7cdb9917467ebc03e7890a98794d P 40c4cc7b32f14b0e89a6d4e89511a047 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 983a3b316b1f4809ae803c17f6c0def5 in term 1.
I20260927 03:58:52.379954 23713 leader_election.cc:302] T 89bd7cdb9917467ebc03e7890a98794d P 983a3b316b1f4809ae803c17f6c0def5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c4cc7b32f14b0e89a6d4e89511a047, 983a3b316b1f4809ae803c17f6c0def5; no voters:
I20260927 03:58:52.379982 24059 raft_consensus.cc:2454] T 89bd7cdb9917467ebc03e7890a98794d P 94088c272ea94c3f909bff38cd2bfbfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 983a3b316b1f4809ae803c17f6c0def5 in term 1.
I20260927 03:58:52.380035 24164 raft_consensus.cc:2781] T 89bd7cdb9917467ebc03e7890a98794d P 983a3b316b1f4809ae803c17f6c0def5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:58:52.380081 24164 raft_consensus.cc:686] T 89bd7cdb9917467ebc03e7890a98794d P 983a3b316b1f4809ae803c17f6c0def5 [term 1 LEADER]: Becoming Leader. State: Replica: 983a3b316b1f4809ae803c17f6c0def5, State: Running, Role: LEADER
I20260927 03:58:52.380138 24164 consensus_queue.cc:237] T 89bd7cdb9917467ebc03e7890a98794d P 983a3b316b1f4809ae803c17f6c0def5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983a3b316b1f4809ae803c17f6c0def5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46859 } } peers { permanent_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38731 } } peers { permanent_uuid: "94088c272ea94c3f909bff38cd2bfbfb" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36625 } }
I20260927 03:58:52.380487 24141 tablet_replica.cc:439] T 89bd7cdb9917467ebc03e7890a98794d P 983a3b316b1f4809ae803c17f6c0def5: TxnStatusTablet state changed. Reason: New leader 983a3b316b1f4809ae803c17f6c0def5. Latest consensus state: current_term: 1 leader_uuid: "983a3b316b1f4809ae803c17f6c0def5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983a3b316b1f4809ae803c17f6c0def5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46859 } } peers { permanent_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38731 } } peers { permanent_uuid: "94088c272ea94c3f909bff38cd2bfbfb" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36625 } } }
I20260927 03:58:52.380561 24141 tablet_replica.cc:442] T 89bd7cdb9917467ebc03e7890a98794d P 983a3b316b1f4809ae803c17f6c0def5: This TxnStatusTablet replica's current role is: LEADER
I20260927 03:58:52.380765 24176 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 03:58:52.380892 24176 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 03:58:52.380823 23621 catalog_manager.cc:5495] T 89bd7cdb9917467ebc03e7890a98794d P 983a3b316b1f4809ae803c17f6c0def5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 983a3b316b1f4809ae803c17f6c0def5 (127.15.185.65). New cstate: current_term: 1 leader_uuid: "983a3b316b1f4809ae803c17f6c0def5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983a3b316b1f4809ae803c17f6c0def5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94088c272ea94c3f909bff38cd2bfbfb" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36625 } health_report { overall_health: UNKNOWN } } }
I20260927 03:58:52.400084 24171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:52.396877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.65:46859 (local address 127.0.0.1:45590)
0927 03:58:52.396986 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:52.396998 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:52.397047 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:52.397366 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:52.397367 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:52.397368 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 03:58:52.397416 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:52.397418 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:52.398399 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:52.398400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:52.399023 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:52.399025 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:52.399552 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:52.399553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:52.399600 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:52.399618 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 03:58:52.399934 (+ 316us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 03:58:52.399941 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 03:58:52.399977 (+ 36us) client_negotiation.cc:240] Negotiation successful
0927 03:58:52.400017 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 03:58:52.400117 24131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:52.397142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45590 (local address 127.15.185.65:46859)
0927 03:58:52.397208 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:52.397211 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:52.397220 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:58:52.397255 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:52.397256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:52.397278 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:52.397355 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:58:52.397484 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:52.398297 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:52.399168 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:52.399480 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:52.399556 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:52.399710 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:58:52.399846 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:58:52.399920 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:52.399999 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 03:58:52.400020 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 03:58:52.401290 23919 raft_consensus.cc:1261] T 89bd7cdb9917467ebc03e7890a98794d P 40c4cc7b32f14b0e89a6d4e89511a047 [term 1 FOLLOWER]: Refusing update from remote peer 983a3b316b1f4809ae803c17f6c0def5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:58:52.401326 24059 raft_consensus.cc:1261] T 89bd7cdb9917467ebc03e7890a98794d P 94088c272ea94c3f909bff38cd2bfbfb [term 1 FOLLOWER]: Refusing update from remote peer 983a3b316b1f4809ae803c17f6c0def5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:58:52.401569 24141 consensus_queue.cc:1035] T 89bd7cdb9917467ebc03e7890a98794d P 983a3b316b1f4809ae803c17f6c0def5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:58:52.401698 24164 consensus_queue.cc:1035] T 89bd7cdb9917467ebc03e7890a98794d P 983a3b316b1f4809ae803c17f6c0def5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94088c272ea94c3f909bff38cd2bfbfb" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:58:52.402366 24151 tablet_replica.cc:439] T 89bd7cdb9917467ebc03e7890a98794d P 94088c272ea94c3f909bff38cd2bfbfb: TxnStatusTablet state changed. Reason: New leader 983a3b316b1f4809ae803c17f6c0def5. Latest consensus state: current_term: 1 leader_uuid: "983a3b316b1f4809ae803c17f6c0def5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983a3b316b1f4809ae803c17f6c0def5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46859 } } peers { permanent_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38731 } } peers { permanent_uuid: "94088c272ea94c3f909bff38cd2bfbfb" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36625 } } }
I20260927 03:58:52.402413 24140 tablet_replica.cc:439] T 89bd7cdb9917467ebc03e7890a98794d P 40c4cc7b32f14b0e89a6d4e89511a047: TxnStatusTablet state changed. Reason: New leader 983a3b316b1f4809ae803c17f6c0def5. Latest consensus state: current_term: 1 leader_uuid: "983a3b316b1f4809ae803c17f6c0def5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983a3b316b1f4809ae803c17f6c0def5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46859 } } peers { permanent_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38731 } } peers { permanent_uuid: "94088c272ea94c3f909bff38cd2bfbfb" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36625 } } }
I20260927 03:58:52.402474 24151 tablet_replica.cc:442] T 89bd7cdb9917467ebc03e7890a98794d P 94088c272ea94c3f909bff38cd2bfbfb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:58:52.402495 24140 tablet_replica.cc:442] T 89bd7cdb9917467ebc03e7890a98794d P 40c4cc7b32f14b0e89a6d4e89511a047: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:58:52.403218 24164 tablet_replica.cc:439] T 89bd7cdb9917467ebc03e7890a98794d P 983a3b316b1f4809ae803c17f6c0def5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "983a3b316b1f4809ae803c17f6c0def5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983a3b316b1f4809ae803c17f6c0def5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46859 } } peers { permanent_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38731 } } peers { permanent_uuid: "94088c272ea94c3f909bff38cd2bfbfb" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36625 } } }
I20260927 03:58:52.403299 24164 tablet_replica.cc:442] T 89bd7cdb9917467ebc03e7890a98794d P 983a3b316b1f4809ae803c17f6c0def5: This TxnStatusTablet replica's current role is: LEADER
I20260927 03:58:52.403358 24164 tablet_replica.cc:439] T 89bd7cdb9917467ebc03e7890a98794d P 983a3b316b1f4809ae803c17f6c0def5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "983a3b316b1f4809ae803c17f6c0def5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983a3b316b1f4809ae803c17f6c0def5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46859 } } peers { permanent_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38731 } } peers { permanent_uuid: "94088c272ea94c3f909bff38cd2bfbfb" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36625 } } }
I20260927 03:58:52.403395 24164 tablet_replica.cc:442] T 89bd7cdb9917467ebc03e7890a98794d P 983a3b316b1f4809ae803c17f6c0def5: This TxnStatusTablet replica's current role is: LEADER
I20260927 03:58:52.403720 24151 tablet_replica.cc:439] T 89bd7cdb9917467ebc03e7890a98794d P 94088c272ea94c3f909bff38cd2bfbfb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "983a3b316b1f4809ae803c17f6c0def5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983a3b316b1f4809ae803c17f6c0def5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46859 } } peers { permanent_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38731 } } peers { permanent_uuid: "94088c272ea94c3f909bff38cd2bfbfb" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36625 } } }
I20260927 03:58:52.403807 24151 tablet_replica.cc:442] T 89bd7cdb9917467ebc03e7890a98794d P 94088c272ea94c3f909bff38cd2bfbfb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:58:52.404078 24140 tablet_replica.cc:439] T 89bd7cdb9917467ebc03e7890a98794d P 40c4cc7b32f14b0e89a6d4e89511a047: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "983a3b316b1f4809ae803c17f6c0def5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983a3b316b1f4809ae803c17f6c0def5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46859 } } peers { permanent_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38731 } } peers { permanent_uuid: "94088c272ea94c3f909bff38cd2bfbfb" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36625 } } }
I20260927 03:58:52.404167 24140 tablet_replica.cc:442] T 89bd7cdb9917467ebc03e7890a98794d P 40c4cc7b32f14b0e89a6d4e89511a047: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 03:58:52.404085 24148 mvcc.cc:205] Tried to move back new op lower bound from 7333812356713517056 to 7333812356629487616. Current Snapshot: MvccSnapshot[applied={T|T < 7333812356713517056}]
I20260927 03:58:52.407157 24131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:52.404953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45598 (local address 127.15.185.65:46859)
0927 03:58:52.404986 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:52.404988 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:52.404995 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:58:52.405015 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:52.405015 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:52.405024 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:52.405073 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:58:52.405215 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:52.405693 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:52.406436 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:52.406641 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:52.406724 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:52.406879 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:58:52.406913 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:58:52.406977 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:52.407038 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 03:58:52.407078 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 03:58:52.412457 24131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:52.409644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:48615 (local address 127.15.185.65:46859)
0927 03:58:52.409686 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:52.409688 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:52.409823 (+ 135us) server_negotiation.cc:407] Connection header received
0927 03:58:52.409876 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:52.409876 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:52.409886 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:52.409943 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:58:52.410082 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:52.410575 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:52.411727 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:52.411905 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:52.411968 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:52.412189 (+ 221us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:58:52.412221 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:58:52.412279 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:52.412344 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 03:58:52.412363 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 03:58:52.412721 24184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:52.409553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.65:46859 (local address 127.15.185.65:48615)
0927 03:58:52.409746 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:52.409757 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:52.409818 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:52.409956 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:52.409956 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:52.409958 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 03:58:52.410019 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:52.410020 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:52.410644 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:52.410644 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:52.411646 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:52.411649 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:52.411993 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:52.411994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:52.412058 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:52.412072 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 03:58:52.412295 (+ 223us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 03:58:52.412297 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 03:58:52.412344 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 03:58:52.412389 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":1,"thread_start_us":105,"threads_started":1}
I20260927 03:58:52.414866 24059 raft_consensus.cc:1261] T 486f35c1d64e40459f1db6df0d1faff1 P 94088c272ea94c3f909bff38cd2bfbfb [term 1 FOLLOWER]: Refusing update from remote peer 983a3b316b1f4809ae803c17f6c0def5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:58:52.414913 23919 raft_consensus.cc:1261] T 486f35c1d64e40459f1db6df0d1faff1 P 40c4cc7b32f14b0e89a6d4e89511a047 [term 1 FOLLOWER]: Refusing update from remote peer 983a3b316b1f4809ae803c17f6c0def5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:58:52.415091 24141 consensus_queue.cc:1035] T 486f35c1d64e40459f1db6df0d1faff1 P 983a3b316b1f4809ae803c17f6c0def5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94088c272ea94c3f909bff38cd2bfbfb" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:58:52.415174 24164 consensus_queue.cc:1035] T 486f35c1d64e40459f1db6df0d1faff1 P 983a3b316b1f4809ae803c17f6c0def5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:58:52.424180 24146 mvcc.cc:205] Tried to move back new op lower bound from 7333812356769988608 to 7333812356191391744. Current Snapshot: MvccSnapshot[applied={T|T < 7333812356769988608}]
I20260927 03:58:52.430840 23621 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57228:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:58:52.430956 23621 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57228:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:58:52.431650 23621 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 523c89766816419d81500ea2272b97f5: Sending DeleteTablet for 3 replicas of tablet 486f35c1d64e40459f1db6df0d1faff1
I20260927 03:58:52.431895 24039 tablet_service.cc:1514] Processing DeleteTablet for tablet 486f35c1d64e40459f1db6df0d1faff1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:58:52 UTC) from {username='oryx', principal='oryx/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:51824
I20260927 03:58:52.431913 23759 tablet_service.cc:1514] Processing DeleteTablet for tablet 486f35c1d64e40459f1db6df0d1faff1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:58:52 UTC) from {username='oryx', principal='oryx/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:45574
I20260927 03:58:52.431885 23899 tablet_service.cc:1514] Processing DeleteTablet for tablet 486f35c1d64e40459f1db6df0d1faff1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:58:52 UTC) from {username='oryx', principal='oryx/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:37984
I20260927 03:58:52.432029 24156 tablet_replica.cc:330] T 486f35c1d64e40459f1db6df0d1faff1 P 94088c272ea94c3f909bff38cd2bfbfb: stopping tablet replica
I20260927 03:58:52.432035 24157 tablet_replica.cc:330] T 486f35c1d64e40459f1db6df0d1faff1 P 983a3b316b1f4809ae803c17f6c0def5: stopping tablet replica
I20260927 03:58:52.432085 24157 raft_consensus.cc:2229] T 486f35c1d64e40459f1db6df0d1faff1 P 983a3b316b1f4809ae803c17f6c0def5 [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:58:52.432094 24156 raft_consensus.cc:2229] T 486f35c1d64e40459f1db6df0d1faff1 P 94088c272ea94c3f909bff38cd2bfbfb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:58:52.432142 24156 raft_consensus.cc:2258] T 486f35c1d64e40459f1db6df0d1faff1 P 94088c272ea94c3f909bff38cd2bfbfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:58:52.432174 24157 raft_consensus.cc:2258] T 486f35c1d64e40459f1db6df0d1faff1 P 983a3b316b1f4809ae803c17f6c0def5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:58:52.432221 24158 tablet_replica.cc:330] T 486f35c1d64e40459f1db6df0d1faff1 P 40c4cc7b32f14b0e89a6d4e89511a047: stopping tablet replica
I20260927 03:58:52.432282 24158 raft_consensus.cc:2229] T 486f35c1d64e40459f1db6df0d1faff1 P 40c4cc7b32f14b0e89a6d4e89511a047 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:58:52.432320 24158 raft_consensus.cc:2258] T 486f35c1d64e40459f1db6df0d1faff1 P 40c4cc7b32f14b0e89a6d4e89511a047 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:58:52.432453 24157 ts_tablet_manager.cc:1905] T 486f35c1d64e40459f1db6df0d1faff1 P 983a3b316b1f4809ae803c17f6c0def5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:58:52.434377 24157 ts_tablet_manager.cc:1918] T 486f35c1d64e40459f1db6df0d1faff1 P 983a3b316b1f4809ae803c17f6c0def5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:58:52.434504 24157 log.cc:1196] T 486f35c1d64e40459f1db6df0d1faff1 P 983a3b316b1f4809ae803c17f6c0def5: Deleting WAL directory at /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/ts-0/wal/wals/486f35c1d64e40459f1db6df0d1faff1
I20260927 03:58:52.434724 24156 ts_tablet_manager.cc:1905] T 486f35c1d64e40459f1db6df0d1faff1 P 94088c272ea94c3f909bff38cd2bfbfb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:58:52.434748 24157 ts_tablet_manager.cc:1939] T 486f35c1d64e40459f1db6df0d1faff1 P 983a3b316b1f4809ae803c17f6c0def5: Deleting consensus metadata
I20260927 03:58:52.435187 23606 catalog_manager.cc:4841] TS 983a3b316b1f4809ae803c17f6c0def5 (127.15.185.65:46859): tablet 486f35c1d64e40459f1db6df0d1faff1 (table test-table [id=2bde981f3670455e8b976b758a0ed989]) successfully deleted
I20260927 03:58:52.436412 24156 ts_tablet_manager.cc:1918] T 486f35c1d64e40459f1db6df0d1faff1 P 94088c272ea94c3f909bff38cd2bfbfb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:58:52.436474 24156 log.cc:1196] T 486f35c1d64e40459f1db6df0d1faff1 P 94088c272ea94c3f909bff38cd2bfbfb: Deleting WAL directory at /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/ts-2/wal/wals/486f35c1d64e40459f1db6df0d1faff1
I20260927 03:58:52.436543 24158 ts_tablet_manager.cc:1905] T 486f35c1d64e40459f1db6df0d1faff1 P 40c4cc7b32f14b0e89a6d4e89511a047: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:58:52.436661 24156 ts_tablet_manager.cc:1939] T 486f35c1d64e40459f1db6df0d1faff1 P 94088c272ea94c3f909bff38cd2bfbfb: Deleting consensus metadata
I20260927 03:58:52.436967 23607 catalog_manager.cc:4841] TS 94088c272ea94c3f909bff38cd2bfbfb (127.15.185.67:36625): tablet 486f35c1d64e40459f1db6df0d1faff1 (table test-table [id=2bde981f3670455e8b976b758a0ed989]) successfully deleted
I20260927 03:58:52.437971 24158 ts_tablet_manager.cc:1918] T 486f35c1d64e40459f1db6df0d1faff1 P 40c4cc7b32f14b0e89a6d4e89511a047: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:58:52.438016 24158 log.cc:1196] T 486f35c1d64e40459f1db6df0d1faff1 P 40c4cc7b32f14b0e89a6d4e89511a047: Deleting WAL directory at /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/ts-1/wal/wals/486f35c1d64e40459f1db6df0d1faff1
I20260927 03:58:52.438202 24158 ts_tablet_manager.cc:1939] T 486f35c1d64e40459f1db6df0d1faff1 P 40c4cc7b32f14b0e89a6d4e89511a047: Deleting consensus metadata
I20260927 03:58:52.438635 23609 catalog_manager.cc:4841] TS 40c4cc7b32f14b0e89a6d4e89511a047 (127.15.185.66:38731): tablet 486f35c1d64e40459f1db6df0d1faff1 (table test-table [id=2bde981f3670455e8b976b758a0ed989]) successfully deleted
Sep 27 03:58:52 dist-test-slave-glgz krb5kdc[23577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 03:58:52 dist-test-slave-glgz krb5kdc[23577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:58:52 dist-test-slave-glgz krb5kdc[23577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
W20260927 03:58:52.462494 23681 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:58:52.455888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57250 (local address 127.15.185.126:34491)
0927 03:58:52.455964 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:52.455966 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:52.456073 (+ 107us) server_negotiation.cc:407] Connection header received
0927 03:58:52.456130 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:52.456131 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:52.456144 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:52.456223 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:52.456909 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:52.457511 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:52.458505 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:52.458810 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:52.458889 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:52.462427 (+ 3538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57250: BlockingRecv error: recv got EOF from 127.0.0.1:57250 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 27 03:58:52 dist-test-slave-glgz krb5kdc[23577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM
I20260927 03:58:52.471280 23681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:52.463476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57260 (local address 127.15.185.126:34491)
0927 03:58:52.463544 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:52.463547 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:52.463607 (+ 60us) server_negotiation.cc:407] Connection header received
0927 03:58:52.463692 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:52.463693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:52.463703 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:52.463766 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:52.464305 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:52.464826 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:52.465644 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:52.465885 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:52.465983 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:52.468416 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:52.468424 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:52.468424 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:52.468432 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:52.469991 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:52.470397 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:52.470399 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:52.470399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:52.470460 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:52.470693 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:52.470694 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:52.470694 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:52.470825 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:52.470943 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:52.471073 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 03:58:52.471160 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 03:58:52.473021 23621 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57260:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:58:52.473205 23621 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:58:52.474782 23759 tablet_service.cc:1467] Processing CreateTablet for tablet 2c93d3cb302c4a9faa6f2f08f05927e2 (DEFAULT_TABLE table=test-table [id=36c128b1bf854596b47d9fcc3f00f634]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:58:52.474810 24039 tablet_service.cc:1467] Processing CreateTablet for tablet 2c93d3cb302c4a9faa6f2f08f05927e2 (DEFAULT_TABLE table=test-table [id=36c128b1bf854596b47d9fcc3f00f634]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:58:52.474879 23899 tablet_service.cc:1467] Processing CreateTablet for tablet 2c93d3cb302c4a9faa6f2f08f05927e2 (DEFAULT_TABLE table=test-table [id=36c128b1bf854596b47d9fcc3f00f634]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:58:52.474921 23759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c93d3cb302c4a9faa6f2f08f05927e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:52.474928 24039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c93d3cb302c4a9faa6f2f08f05927e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:52.475008 23899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c93d3cb302c4a9faa6f2f08f05927e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:52.476406 24135 tablet_bootstrap.cc:492] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 983a3b316b1f4809ae803c17f6c0def5: Bootstrap starting.
I20260927 03:58:52.476838 24135 tablet_bootstrap.cc:654] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 983a3b316b1f4809ae803c17f6c0def5: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:52.477259 24134 tablet_bootstrap.cc:492] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 94088c272ea94c3f909bff38cd2bfbfb: Bootstrap starting.
I20260927 03:58:52.477607 24134 tablet_bootstrap.cc:654] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 94088c272ea94c3f909bff38cd2bfbfb: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:52.477867 24133 tablet_bootstrap.cc:492] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 40c4cc7b32f14b0e89a6d4e89511a047: Bootstrap starting.
I20260927 03:58:52.478257 24133 tablet_bootstrap.cc:654] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 40c4cc7b32f14b0e89a6d4e89511a047: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:52.478812 24134 tablet_bootstrap.cc:492] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 94088c272ea94c3f909bff38cd2bfbfb: No bootstrap required, opened a new log
I20260927 03:58:52.478869 24134 ts_tablet_manager.cc:1397] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 94088c272ea94c3f909bff38cd2bfbfb: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 03:58:52.478976 24134 raft_consensus.cc:348] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 94088c272ea94c3f909bff38cd2bfbfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983a3b316b1f4809ae803c17f6c0def5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46859 } } peers { permanent_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38731 } } peers { permanent_uuid: "94088c272ea94c3f909bff38cd2bfbfb" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36625 } }
I20260927 03:58:52.479079 24134 raft_consensus.cc:374] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 94088c272ea94c3f909bff38cd2bfbfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:52.479104 24134 raft_consensus.cc:729] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 94088c272ea94c3f909bff38cd2bfbfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94088c272ea94c3f909bff38cd2bfbfb, State: Initialized, Role: FOLLOWER
I20260927 03:58:52.479171 24134 consensus_queue.cc:260] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 94088c272ea94c3f909bff38cd2bfbfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983a3b316b1f4809ae803c17f6c0def5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46859 } } peers { permanent_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38731 } } peers { permanent_uuid: "94088c272ea94c3f909bff38cd2bfbfb" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36625 } }
I20260927 03:58:52.479269 24134 ts_tablet_manager.cc:1428] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 94088c272ea94c3f909bff38cd2bfbfb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:52.479614 24133 tablet_bootstrap.cc:492] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 40c4cc7b32f14b0e89a6d4e89511a047: No bootstrap required, opened a new log
I20260927 03:58:52.479664 24133 ts_tablet_manager.cc:1397] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 40c4cc7b32f14b0e89a6d4e89511a047: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:58:52.479837 24133 raft_consensus.cc:348] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 40c4cc7b32f14b0e89a6d4e89511a047 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983a3b316b1f4809ae803c17f6c0def5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46859 } } peers { permanent_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38731 } } peers { permanent_uuid: "94088c272ea94c3f909bff38cd2bfbfb" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36625 } }
I20260927 03:58:52.479920 24133 raft_consensus.cc:374] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 40c4cc7b32f14b0e89a6d4e89511a047 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:52.479941 24133 raft_consensus.cc:729] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 40c4cc7b32f14b0e89a6d4e89511a047 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c4cc7b32f14b0e89a6d4e89511a047, State: Initialized, Role: FOLLOWER
I20260927 03:58:52.479982 24133 consensus_queue.cc:260] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 40c4cc7b32f14b0e89a6d4e89511a047 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983a3b316b1f4809ae803c17f6c0def5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46859 } } peers { permanent_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38731 } } peers { permanent_uuid: "94088c272ea94c3f909bff38cd2bfbfb" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36625 } }
I20260927 03:58:52.480095 24133 ts_tablet_manager.cc:1428] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 40c4cc7b32f14b0e89a6d4e89511a047: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 03:58:52.480405 24135 tablet_bootstrap.cc:492] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 983a3b316b1f4809ae803c17f6c0def5: No bootstrap required, opened a new log
I20260927 03:58:52.480455 24135 ts_tablet_manager.cc:1397] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 983a3b316b1f4809ae803c17f6c0def5: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260927 03:58:52.480638 24135 raft_consensus.cc:348] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 983a3b316b1f4809ae803c17f6c0def5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983a3b316b1f4809ae803c17f6c0def5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46859 } } peers { permanent_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38731 } } peers { permanent_uuid: "94088c272ea94c3f909bff38cd2bfbfb" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36625 } }
I20260927 03:58:52.480739 24135 raft_consensus.cc:374] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 983a3b316b1f4809ae803c17f6c0def5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:52.480762 24135 raft_consensus.cc:729] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 983a3b316b1f4809ae803c17f6c0def5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 983a3b316b1f4809ae803c17f6c0def5, State: Initialized, Role: FOLLOWER
I20260927 03:58:52.480803 24135 consensus_queue.cc:260] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 983a3b316b1f4809ae803c17f6c0def5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983a3b316b1f4809ae803c17f6c0def5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46859 } } peers { permanent_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38731 } } peers { permanent_uuid: "94088c272ea94c3f909bff38cd2bfbfb" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36625 } }
I20260927 03:58:52.480903 24135 ts_tablet_manager.cc:1428] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 983a3b316b1f4809ae803c17f6c0def5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:52.550731 24140 raft_consensus.cc:482] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 40c4cc7b32f14b0e89a6d4e89511a047 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:58:52.550848 24140 raft_consensus.cc:504] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 40c4cc7b32f14b0e89a6d4e89511a047 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983a3b316b1f4809ae803c17f6c0def5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46859 } } peers { permanent_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38731 } } peers { permanent_uuid: "94088c272ea94c3f909bff38cd2bfbfb" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36625 } }
I20260927 03:58:52.551235 24140 leader_election.cc:288] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 40c4cc7b32f14b0e89a6d4e89511a047 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 983a3b316b1f4809ae803c17f6c0def5 (127.15.185.65:46859), 94088c272ea94c3f909bff38cd2bfbfb (127.15.185.67:36625)
I20260927 03:58:52.555202 24205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:52.551405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.67:36625 (local address 127.15.185.66:33933)
0927 03:58:52.551675 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:52.551691 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:52.551741 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:52.551997 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:52.551998 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:52.552003 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:58:52.552123 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:52.552125 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:52.552785 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:52.552786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:52.554132 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:52.554135 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:52.554902 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:52.554904 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:52.554971 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:52.554976 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 03:58:52.555031 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 03:58:52.555097 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":22,"thread_start_us":87,"threads_started":1}
I20260927 03:58:52.555219 24132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:52.551603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:33933 (local address 127.15.185.67:36625)
0927 03:58:52.551653 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:52.551655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:52.551723 (+ 68us) server_negotiation.cc:407] Connection header received
0927 03:58:52.551806 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:52.551807 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:52.551818 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:52.551868 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:58:52.552230 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:52.552700 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:52.554223 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:52.554789 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:52.554868 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:52.554876 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:52.555082 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 03:58:52.555125 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 03:58:52.555266 24131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:52.551472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:53915 (local address 127.15.185.65:46859)
0927 03:58:52.551523 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:52.551525 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:52.551642 (+ 117us) server_negotiation.cc:407] Connection header received
0927 03:58:52.551901 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:52.551902 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:52.551912 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:52.551977 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:58:52.552207 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:52.552934 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:52.554468 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:52.554866 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:52.554943 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:52.554952 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:52.555152 (+ 200us) server_negotiation.cc:299] Negotiation successful
0927 03:58:52.555191 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 03:58:52.555482 24058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c93d3cb302c4a9faa6f2f08f05927e2" candidate_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94088c272ea94c3f909bff38cd2bfbfb" is_pre_election: true
I20260927 03:58:52.555581 24058 raft_consensus.cc:2454] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 94088c272ea94c3f909bff38cd2bfbfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40c4cc7b32f14b0e89a6d4e89511a047 in term 0.
I20260927 03:58:52.555651 24206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:52.551405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.65:46859 (local address 127.15.185.66:53915)
0927 03:58:52.551575 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:52.551586 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:52.551809 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:52.551997 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:52.551998 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:52.552001 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:58:52.552130 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:52.552132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:52.553020 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:52.553021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:52.554392 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:52.554394 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:52.554982 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:52.554983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:52.555072 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:52.555075 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:58:52.555501 (+ 426us) client_negotiation.cc:240] Negotiation successful
0927 03:58:52.555540 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":23,"thread_start_us":85,"threads_started":1}
I20260927 03:58:52.555751 23851 leader_election.cc:302] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 40c4cc7b32f14b0e89a6d4e89511a047 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c4cc7b32f14b0e89a6d4e89511a047, 94088c272ea94c3f909bff38cd2bfbfb; no voters:
I20260927 03:58:52.555876 24140 raft_consensus.cc:2781] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 40c4cc7b32f14b0e89a6d4e89511a047 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:58:52.555927 24140 raft_consensus.cc:482] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 40c4cc7b32f14b0e89a6d4e89511a047 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:58:52.555955 24140 raft_consensus.cc:3037] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 40c4cc7b32f14b0e89a6d4e89511a047 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:52.555922 23779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c93d3cb302c4a9faa6f2f08f05927e2" candidate_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "983a3b316b1f4809ae803c17f6c0def5" is_pre_election: true
I20260927 03:58:52.556003 23779 raft_consensus.cc:2454] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 983a3b316b1f4809ae803c17f6c0def5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40c4cc7b32f14b0e89a6d4e89511a047 in term 0.
I20260927 03:58:52.556615 24140 raft_consensus.cc:504] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 40c4cc7b32f14b0e89a6d4e89511a047 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983a3b316b1f4809ae803c17f6c0def5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46859 } } peers { permanent_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38731 } } peers { permanent_uuid: "94088c272ea94c3f909bff38cd2bfbfb" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36625 } }
I20260927 03:58:52.556756 24140 leader_election.cc:288] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 40c4cc7b32f14b0e89a6d4e89511a047 [CANDIDATE]: Term 1 election: Requested vote from peers 983a3b316b1f4809ae803c17f6c0def5 (127.15.185.65:46859), 94088c272ea94c3f909bff38cd2bfbfb (127.15.185.67:36625)
I20260927 03:58:52.556891 23779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c93d3cb302c4a9faa6f2f08f05927e2" candidate_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "983a3b316b1f4809ae803c17f6c0def5"
I20260927 03:58:52.556972 23779 raft_consensus.cc:3037] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 983a3b316b1f4809ae803c17f6c0def5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:52.557148 24058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c93d3cb302c4a9faa6f2f08f05927e2" candidate_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94088c272ea94c3f909bff38cd2bfbfb"
I20260927 03:58:52.557220 24058 raft_consensus.cc:3037] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 94088c272ea94c3f909bff38cd2bfbfb [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:52.557727 23779 raft_consensus.cc:2454] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 983a3b316b1f4809ae803c17f6c0def5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40c4cc7b32f14b0e89a6d4e89511a047 in term 1.
I20260927 03:58:52.557904 23850 leader_election.cc:302] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 40c4cc7b32f14b0e89a6d4e89511a047 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c4cc7b32f14b0e89a6d4e89511a047, 983a3b316b1f4809ae803c17f6c0def5; no voters:
I20260927 03:58:52.558028 24140 raft_consensus.cc:2781] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 40c4cc7b32f14b0e89a6d4e89511a047 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:58:52.558089 24058 raft_consensus.cc:2454] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 94088c272ea94c3f909bff38cd2bfbfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40c4cc7b32f14b0e89a6d4e89511a047 in term 1.
I20260927 03:58:52.558166 24140 raft_consensus.cc:686] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 40c4cc7b32f14b0e89a6d4e89511a047 [term 1 LEADER]: Becoming Leader. State: Replica: 40c4cc7b32f14b0e89a6d4e89511a047, State: Running, Role: LEADER
I20260927 03:58:52.558233 24140 consensus_queue.cc:237] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 40c4cc7b32f14b0e89a6d4e89511a047 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983a3b316b1f4809ae803c17f6c0def5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46859 } } peers { permanent_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38731 } } peers { permanent_uuid: "94088c272ea94c3f909bff38cd2bfbfb" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36625 } }
I20260927 03:58:52.558992 23620 catalog_manager.cc:5495] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 40c4cc7b32f14b0e89a6d4e89511a047 reported cstate change: term changed from 0 to 1, leader changed from <none> to 40c4cc7b32f14b0e89a6d4e89511a047 (127.15.185.66). New cstate: current_term: 1 leader_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983a3b316b1f4809ae803c17f6c0def5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "94088c272ea94c3f909bff38cd2bfbfb" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36625 } health_report { overall_health: UNKNOWN } } }
I20260927 03:58:52.576933 24130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:52.574031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38000 (local address 127.15.185.66:38731)
0927 03:58:52.574073 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:52.574075 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:52.574108 (+ 33us) server_negotiation.cc:407] Connection header received
0927 03:58:52.574200 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:52.574201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:52.574229 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:52.574325 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:58:52.574496 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:52.575311 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:52.576097 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:52.576398 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:52.576468 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:52.576579 (+ 111us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:58:52.576724 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:58:52.576800 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:52.576830 (+ 30us) server_negotiation.cc:299] Negotiation successful
0927 03:58:52.576851 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 03:58:52.578056 24058 raft_consensus.cc:1261] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 94088c272ea94c3f909bff38cd2bfbfb [term 1 FOLLOWER]: Refusing update from remote peer 40c4cc7b32f14b0e89a6d4e89511a047: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:58:52.578078 23779 raft_consensus.cc:1261] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 983a3b316b1f4809ae803c17f6c0def5 [term 1 FOLLOWER]: Refusing update from remote peer 40c4cc7b32f14b0e89a6d4e89511a047: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:58:52.578382 24207 consensus_queue.cc:1035] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 40c4cc7b32f14b0e89a6d4e89511a047 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94088c272ea94c3f909bff38cd2bfbfb" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:58:52.578562 24140 consensus_queue.cc:1035] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 40c4cc7b32f14b0e89a6d4e89511a047 [LEADER]: Connected to new peer: Peer: permanent_uuid: "983a3b316b1f4809ae803c17f6c0def5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:58:52.582094 23620 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57260:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:58:52.582198 23620 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57260:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:58:52.582970 23620 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 523c89766816419d81500ea2272b97f5: Sending DeleteTablet for 3 replicas of tablet 2c93d3cb302c4a9faa6f2f08f05927e2
I20260927 03:58:52.583216 23899 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c93d3cb302c4a9faa6f2f08f05927e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:58:52 UTC) from {username='oryx', principal='oryx/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:37984
I20260927 03:58:52.583262 23759 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c93d3cb302c4a9faa6f2f08f05927e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:58:52 UTC) from {username='oryx', principal='oryx/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:45574
I20260927 03:58:52.583362 24158 tablet_replica.cc:330] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 40c4cc7b32f14b0e89a6d4e89511a047: stopping tablet replica
I20260927 03:58:52.583438 24158 raft_consensus.cc:2229] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 40c4cc7b32f14b0e89a6d4e89511a047 [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:58:52.583532 24158 raft_consensus.cc:2258] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 40c4cc7b32f14b0e89a6d4e89511a047 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:58:52.583616 24157 tablet_replica.cc:330] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 983a3b316b1f4809ae803c17f6c0def5: stopping tablet replica
I20260927 03:58:52.583673 24157 raft_consensus.cc:2229] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 983a3b316b1f4809ae803c17f6c0def5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:58:52.583720 24157 raft_consensus.cc:2258] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 983a3b316b1f4809ae803c17f6c0def5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:58:52.584043 24039 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c93d3cb302c4a9faa6f2f08f05927e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:58:52 UTC) from {username='oryx', principal='oryx/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:51824
I20260927 03:58:52.584134 24156 tablet_replica.cc:330] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 94088c272ea94c3f909bff38cd2bfbfb: stopping tablet replica
I20260927 03:58:52.584179 24156 raft_consensus.cc:2229] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 94088c272ea94c3f909bff38cd2bfbfb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:58:52.584219 24156 raft_consensus.cc:2258] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 94088c272ea94c3f909bff38cd2bfbfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:58:52.584281 23620 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57260:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:58:52.584444 23620 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:58:52.584600 24156 ts_tablet_manager.cc:1905] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 94088c272ea94c3f909bff38cd2bfbfb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:58:52.584863 24158 ts_tablet_manager.cc:1905] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 40c4cc7b32f14b0e89a6d4e89511a047: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:58:52.586180 24156 ts_tablet_manager.cc:1918] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 94088c272ea94c3f909bff38cd2bfbfb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:58:52.586256 24156 log.cc:1196] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 94088c272ea94c3f909bff38cd2bfbfb: Deleting WAL directory at /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/ts-2/wal/wals/2c93d3cb302c4a9faa6f2f08f05927e2
I20260927 03:58:52.586321 24158 ts_tablet_manager.cc:1918] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 40c4cc7b32f14b0e89a6d4e89511a047: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:58:52.586369 24158 log.cc:1196] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 40c4cc7b32f14b0e89a6d4e89511a047: Deleting WAL directory at /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/ts-1/wal/wals/2c93d3cb302c4a9faa6f2f08f05927e2
I20260927 03:58:52.586489 24156 ts_tablet_manager.cc:1939] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 94088c272ea94c3f909bff38cd2bfbfb: Deleting consensus metadata
I20260927 03:58:52.586575 24158 ts_tablet_manager.cc:1939] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 40c4cc7b32f14b0e89a6d4e89511a047: Deleting consensus metadata
I20260927 03:58:52.587072 24157 ts_tablet_manager.cc:1905] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 983a3b316b1f4809ae803c17f6c0def5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:58:52.587401 23607 catalog_manager.cc:4841] TS 94088c272ea94c3f909bff38cd2bfbfb (127.15.185.67:36625): tablet 2c93d3cb302c4a9faa6f2f08f05927e2 (table test-table [id=36c128b1bf854596b47d9fcc3f00f634]) successfully deleted
I20260927 03:58:52.587565 23609 catalog_manager.cc:4841] TS 40c4cc7b32f14b0e89a6d4e89511a047 (127.15.185.66:38731): tablet 2c93d3cb302c4a9faa6f2f08f05927e2 (table test-table [id=36c128b1bf854596b47d9fcc3f00f634]) successfully deleted
I20260927 03:58:52.588514 24157 ts_tablet_manager.cc:1918] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 983a3b316b1f4809ae803c17f6c0def5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:58:52.588574 24157 log.cc:1196] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 983a3b316b1f4809ae803c17f6c0def5: Deleting WAL directory at /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/ts-0/wal/wals/2c93d3cb302c4a9faa6f2f08f05927e2
I20260927 03:58:52.588686 24039 tablet_service.cc:1467] Processing CreateTablet for tablet d4d7d21cc8cd4039a57f2963e924aeda (DEFAULT_TABLE table=test-table [id=e01a6f40b46b4fa7b81360f4b84b3ab2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:58:52.588781 24157 ts_tablet_manager.cc:1939] T 2c93d3cb302c4a9faa6f2f08f05927e2 P 983a3b316b1f4809ae803c17f6c0def5: Deleting consensus metadata
I20260927 03:58:52.588785 24039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4d7d21cc8cd4039a57f2963e924aeda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:52.590046 24134 tablet_bootstrap.cc:492] T d4d7d21cc8cd4039a57f2963e924aeda P 94088c272ea94c3f909bff38cd2bfbfb: Bootstrap starting.
I20260927 03:58:52.590354 24134 tablet_bootstrap.cc:654] T d4d7d21cc8cd4039a57f2963e924aeda P 94088c272ea94c3f909bff38cd2bfbfb: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:52.590754 23606 catalog_manager.cc:4841] TS 983a3b316b1f4809ae803c17f6c0def5 (127.15.185.65:46859): tablet 2c93d3cb302c4a9faa6f2f08f05927e2 (table test-table [id=36c128b1bf854596b47d9fcc3f00f634]) successfully deleted
I20260927 03:58:52.590965 23759 tablet_service.cc:1467] Processing CreateTablet for tablet d4d7d21cc8cd4039a57f2963e924aeda (DEFAULT_TABLE table=test-table [id=e01a6f40b46b4fa7b81360f4b84b3ab2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:58:52.591027 23899 tablet_service.cc:1467] Processing CreateTablet for tablet d4d7d21cc8cd4039a57f2963e924aeda (DEFAULT_TABLE table=test-table [id=e01a6f40b46b4fa7b81360f4b84b3ab2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:58:52.591104 23759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4d7d21cc8cd4039a57f2963e924aeda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:52.591150 23899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4d7d21cc8cd4039a57f2963e924aeda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:52.592298 24133 tablet_bootstrap.cc:492] T d4d7d21cc8cd4039a57f2963e924aeda P 40c4cc7b32f14b0e89a6d4e89511a047: Bootstrap starting.
I20260927 03:58:52.592393 24135 tablet_bootstrap.cc:492] T d4d7d21cc8cd4039a57f2963e924aeda P 983a3b316b1f4809ae803c17f6c0def5: Bootstrap starting.
I20260927 03:58:52.592705 24133 tablet_bootstrap.cc:654] T d4d7d21cc8cd4039a57f2963e924aeda P 40c4cc7b32f14b0e89a6d4e89511a047: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:52.592722 24135 tablet_bootstrap.cc:654] T d4d7d21cc8cd4039a57f2963e924aeda P 983a3b316b1f4809ae803c17f6c0def5: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:52.593407 24134 tablet_bootstrap.cc:492] T d4d7d21cc8cd4039a57f2963e924aeda P 94088c272ea94c3f909bff38cd2bfbfb: No bootstrap required, opened a new log
I20260927 03:58:52.593453 24134 ts_tablet_manager.cc:1397] T d4d7d21cc8cd4039a57f2963e924aeda P 94088c272ea94c3f909bff38cd2bfbfb: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260927 03:58:52.593575 24134 raft_consensus.cc:348] T d4d7d21cc8cd4039a57f2963e924aeda P 94088c272ea94c3f909bff38cd2bfbfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983a3b316b1f4809ae803c17f6c0def5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46859 } } peers { permanent_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38731 } } peers { permanent_uuid: "94088c272ea94c3f909bff38cd2bfbfb" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36625 } }
I20260927 03:58:52.593641 24134 raft_consensus.cc:374] T d4d7d21cc8cd4039a57f2963e924aeda P 94088c272ea94c3f909bff38cd2bfbfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:52.593660 24134 raft_consensus.cc:729] T d4d7d21cc8cd4039a57f2963e924aeda P 94088c272ea94c3f909bff38cd2bfbfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94088c272ea94c3f909bff38cd2bfbfb, State: Initialized, Role: FOLLOWER
I20260927 03:58:52.593708 24134 consensus_queue.cc:260] T d4d7d21cc8cd4039a57f2963e924aeda P 94088c272ea94c3f909bff38cd2bfbfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983a3b316b1f4809ae803c17f6c0def5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46859 } } peers { permanent_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38731 } } peers { permanent_uuid: "94088c272ea94c3f909bff38cd2bfbfb" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36625 } }
I20260927 03:58:52.593791 24134 ts_tablet_manager.cc:1428] T d4d7d21cc8cd4039a57f2963e924aeda P 94088c272ea94c3f909bff38cd2bfbfb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:52.594166 24133 tablet_bootstrap.cc:492] T d4d7d21cc8cd4039a57f2963e924aeda P 40c4cc7b32f14b0e89a6d4e89511a047: No bootstrap required, opened a new log
I20260927 03:58:52.594211 24133 ts_tablet_manager.cc:1397] T d4d7d21cc8cd4039a57f2963e924aeda P 40c4cc7b32f14b0e89a6d4e89511a047: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:58:52.594399 24133 raft_consensus.cc:348] T d4d7d21cc8cd4039a57f2963e924aeda P 40c4cc7b32f14b0e89a6d4e89511a047 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983a3b316b1f4809ae803c17f6c0def5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46859 } } peers { permanent_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38731 } } peers { permanent_uuid: "94088c272ea94c3f909bff38cd2bfbfb" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36625 } }
I20260927 03:58:52.594476 24133 raft_consensus.cc:374] T d4d7d21cc8cd4039a57f2963e924aeda P 40c4cc7b32f14b0e89a6d4e89511a047 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:52.594496 24133 raft_consensus.cc:729] T d4d7d21cc8cd4039a57f2963e924aeda P 40c4cc7b32f14b0e89a6d4e89511a047 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c4cc7b32f14b0e89a6d4e89511a047, State: Initialized, Role: FOLLOWER
I20260927 03:58:52.594549 24133 consensus_queue.cc:260] T d4d7d21cc8cd4039a57f2963e924aeda P 40c4cc7b32f14b0e89a6d4e89511a047 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983a3b316b1f4809ae803c17f6c0def5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46859 } } peers { permanent_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38731 } } peers { permanent_uuid: "94088c272ea94c3f909bff38cd2bfbfb" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36625 } }
I20260927 03:58:52.594630 24133 ts_tablet_manager.cc:1428] T d4d7d21cc8cd4039a57f2963e924aeda P 40c4cc7b32f14b0e89a6d4e89511a047: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:52.599241 24135 tablet_bootstrap.cc:492] T d4d7d21cc8cd4039a57f2963e924aeda P 983a3b316b1f4809ae803c17f6c0def5: No bootstrap required, opened a new log
I20260927 03:58:52.599313 24135 ts_tablet_manager.cc:1397] T d4d7d21cc8cd4039a57f2963e924aeda P 983a3b316b1f4809ae803c17f6c0def5: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260927 03:58:52.599433 24135 raft_consensus.cc:348] T d4d7d21cc8cd4039a57f2963e924aeda P 983a3b316b1f4809ae803c17f6c0def5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983a3b316b1f4809ae803c17f6c0def5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46859 } } peers { permanent_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38731 } } peers { permanent_uuid: "94088c272ea94c3f909bff38cd2bfbfb" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36625 } }
I20260927 03:58:52.599537 24135 raft_consensus.cc:374] T d4d7d21cc8cd4039a57f2963e924aeda P 983a3b316b1f4809ae803c17f6c0def5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:52.599558 24135 raft_consensus.cc:729] T d4d7d21cc8cd4039a57f2963e924aeda P 983a3b316b1f4809ae803c17f6c0def5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 983a3b316b1f4809ae803c17f6c0def5, State: Initialized, Role: FOLLOWER
I20260927 03:58:52.599601 24135 consensus_queue.cc:260] T d4d7d21cc8cd4039a57f2963e924aeda P 983a3b316b1f4809ae803c17f6c0def5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983a3b316b1f4809ae803c17f6c0def5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46859 } } peers { permanent_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38731 } } peers { permanent_uuid: "94088c272ea94c3f909bff38cd2bfbfb" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36625 } }
I20260927 03:58:52.599699 24135 ts_tablet_manager.cc:1428] T d4d7d21cc8cd4039a57f2963e924aeda P 983a3b316b1f4809ae803c17f6c0def5: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260927 03:58:52.659205 24141 raft_consensus.cc:482] T d4d7d21cc8cd4039a57f2963e924aeda P 983a3b316b1f4809ae803c17f6c0def5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:58:52.659305 24141 raft_consensus.cc:504] T d4d7d21cc8cd4039a57f2963e924aeda P 983a3b316b1f4809ae803c17f6c0def5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983a3b316b1f4809ae803c17f6c0def5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46859 } } peers { permanent_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38731 } } peers { permanent_uuid: "94088c272ea94c3f909bff38cd2bfbfb" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36625 } }
I20260927 03:58:52.659461 24141 leader_election.cc:288] T d4d7d21cc8cd4039a57f2963e924aeda P 983a3b316b1f4809ae803c17f6c0def5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40c4cc7b32f14b0e89a6d4e89511a047 (127.15.185.66:38731), 94088c272ea94c3f909bff38cd2bfbfb (127.15.185.67:36625)
I20260927 03:58:52.659646 24058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4d7d21cc8cd4039a57f2963e924aeda" candidate_uuid: "983a3b316b1f4809ae803c17f6c0def5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94088c272ea94c3f909bff38cd2bfbfb" is_pre_election: true
I20260927 03:58:52.659708 23919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4d7d21cc8cd4039a57f2963e924aeda" candidate_uuid: "983a3b316b1f4809ae803c17f6c0def5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" is_pre_election: true
I20260927 03:58:52.659754 24058 raft_consensus.cc:2454] T d4d7d21cc8cd4039a57f2963e924aeda P 94088c272ea94c3f909bff38cd2bfbfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 983a3b316b1f4809ae803c17f6c0def5 in term 0.
I20260927 03:58:52.659783 23919 raft_consensus.cc:2454] T d4d7d21cc8cd4039a57f2963e924aeda P 40c4cc7b32f14b0e89a6d4e89511a047 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 983a3b316b1f4809ae803c17f6c0def5 in term 0.
I20260927 03:58:52.659909 23711 leader_election.cc:302] T d4d7d21cc8cd4039a57f2963e924aeda P 983a3b316b1f4809ae803c17f6c0def5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94088c272ea94c3f909bff38cd2bfbfb, 983a3b316b1f4809ae803c17f6c0def5; no voters:
I20260927 03:58:52.660007 24141 raft_consensus.cc:2781] T d4d7d21cc8cd4039a57f2963e924aeda P 983a3b316b1f4809ae803c17f6c0def5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:58:52.660063 24141 raft_consensus.cc:482] T d4d7d21cc8cd4039a57f2963e924aeda P 983a3b316b1f4809ae803c17f6c0def5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:58:52.660085 24141 raft_consensus.cc:3037] T d4d7d21cc8cd4039a57f2963e924aeda P 983a3b316b1f4809ae803c17f6c0def5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:52.660764 24141 raft_consensus.cc:504] T d4d7d21cc8cd4039a57f2963e924aeda P 983a3b316b1f4809ae803c17f6c0def5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983a3b316b1f4809ae803c17f6c0def5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46859 } } peers { permanent_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38731 } } peers { permanent_uuid: "94088c272ea94c3f909bff38cd2bfbfb" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36625 } }
I20260927 03:58:52.660900 24141 leader_election.cc:288] T d4d7d21cc8cd4039a57f2963e924aeda P 983a3b316b1f4809ae803c17f6c0def5 [CANDIDATE]: Term 1 election: Requested vote from peers 40c4cc7b32f14b0e89a6d4e89511a047 (127.15.185.66:38731), 94088c272ea94c3f909bff38cd2bfbfb (127.15.185.67:36625)
I20260927 03:58:52.661021 23919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4d7d21cc8cd4039a57f2963e924aeda" candidate_uuid: "983a3b316b1f4809ae803c17f6c0def5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40c4cc7b32f14b0e89a6d4e89511a047"
I20260927 03:58:52.661103 23919 raft_consensus.cc:3037] T d4d7d21cc8cd4039a57f2963e924aeda P 40c4cc7b32f14b0e89a6d4e89511a047 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:52.661113 24058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4d7d21cc8cd4039a57f2963e924aeda" candidate_uuid: "983a3b316b1f4809ae803c17f6c0def5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94088c272ea94c3f909bff38cd2bfbfb"
I20260927 03:58:52.661222 24058 raft_consensus.cc:3037] T d4d7d21cc8cd4039a57f2963e924aeda P 94088c272ea94c3f909bff38cd2bfbfb [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:52.661922 23919 raft_consensus.cc:2454] T d4d7d21cc8cd4039a57f2963e924aeda P 40c4cc7b32f14b0e89a6d4e89511a047 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 983a3b316b1f4809ae803c17f6c0def5 in term 1.
I20260927 03:58:52.661980 24058 raft_consensus.cc:2454] T d4d7d21cc8cd4039a57f2963e924aeda P 94088c272ea94c3f909bff38cd2bfbfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 983a3b316b1f4809ae803c17f6c0def5 in term 1.
I20260927 03:58:52.662113 23713 leader_election.cc:302] T d4d7d21cc8cd4039a57f2963e924aeda P 983a3b316b1f4809ae803c17f6c0def5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40c4cc7b32f14b0e89a6d4e89511a047, 983a3b316b1f4809ae803c17f6c0def5; no voters:
I20260927 03:58:52.662243 24141 raft_consensus.cc:2781] T d4d7d21cc8cd4039a57f2963e924aeda P 983a3b316b1f4809ae803c17f6c0def5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:58:52.662292 24141 raft_consensus.cc:686] T d4d7d21cc8cd4039a57f2963e924aeda P 983a3b316b1f4809ae803c17f6c0def5 [term 1 LEADER]: Becoming Leader. State: Replica: 983a3b316b1f4809ae803c17f6c0def5, State: Running, Role: LEADER
I20260927 03:58:52.662338 24141 consensus_queue.cc:237] T d4d7d21cc8cd4039a57f2963e924aeda P 983a3b316b1f4809ae803c17f6c0def5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983a3b316b1f4809ae803c17f6c0def5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46859 } } peers { permanent_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38731 } } peers { permanent_uuid: "94088c272ea94c3f909bff38cd2bfbfb" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36625 } }
I20260927 03:58:52.662972 23620 catalog_manager.cc:5495] T d4d7d21cc8cd4039a57f2963e924aeda P 983a3b316b1f4809ae803c17f6c0def5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 983a3b316b1f4809ae803c17f6c0def5 (127.15.185.65). New cstate: current_term: 1 leader_uuid: "983a3b316b1f4809ae803c17f6c0def5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983a3b316b1f4809ae803c17f6c0def5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94088c272ea94c3f909bff38cd2bfbfb" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36625 } health_report { overall_health: UNKNOWN } } }
I20260927 03:58:52.675266 24131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:52.673015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45614 (local address 127.15.185.65:46859)
0927 03:58:52.673071 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:52.673073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:52.673080 (+ 7us) server_negotiation.cc:407] Connection header received
0927 03:58:52.673100 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:52.673100 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:52.673110 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:52.673181 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:58:52.673286 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:52.673773 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:52.674522 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:52.674740 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:52.674816 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:52.674970 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:58:52.675005 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:58:52.675091 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:52.675151 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 03:58:52.675172 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 03:58:52.677556 23919 raft_consensus.cc:1261] T d4d7d21cc8cd4039a57f2963e924aeda P 40c4cc7b32f14b0e89a6d4e89511a047 [term 1 FOLLOWER]: Refusing update from remote peer 983a3b316b1f4809ae803c17f6c0def5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:58:52.677564 24058 raft_consensus.cc:1261] T d4d7d21cc8cd4039a57f2963e924aeda P 94088c272ea94c3f909bff38cd2bfbfb [term 1 FOLLOWER]: Refusing update from remote peer 983a3b316b1f4809ae803c17f6c0def5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:58:52.677747 24141 consensus_queue.cc:1035] T d4d7d21cc8cd4039a57f2963e924aeda P 983a3b316b1f4809ae803c17f6c0def5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40c4cc7b32f14b0e89a6d4e89511a047" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:58:52.677863 24188 consensus_queue.cc:1035] T d4d7d21cc8cd4039a57f2963e924aeda P 983a3b316b1f4809ae803c17f6c0def5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94088c272ea94c3f909bff38cd2bfbfb" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 36625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:58:52.688118 23620 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57260:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:58:52.688234 23620 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57260:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:58:52.688849 23620 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 523c89766816419d81500ea2272b97f5: Sending DeleteTablet for 3 replicas of tablet d4d7d21cc8cd4039a57f2963e924aeda
I20260927 03:58:52.689100 23758 tablet_service.cc:1514] Processing DeleteTablet for tablet d4d7d21cc8cd4039a57f2963e924aeda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:58:52 UTC) from {username='oryx', principal='oryx/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:45574
I20260927 03:58:52.689193 24157 tablet_replica.cc:330] T d4d7d21cc8cd4039a57f2963e924aeda P 983a3b316b1f4809ae803c17f6c0def5: stopping tablet replica
I20260927 03:58:52.689240 24157 raft_consensus.cc:2229] T d4d7d21cc8cd4039a57f2963e924aeda P 983a3b316b1f4809ae803c17f6c0def5 [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:58:52.689301 24157 raft_consensus.cc:2258] T d4d7d21cc8cd4039a57f2963e924aeda P 983a3b316b1f4809ae803c17f6c0def5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:58:52.689379 24039 tablet_service.cc:1514] Processing DeleteTablet for tablet d4d7d21cc8cd4039a57f2963e924aeda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:58:52 UTC) from {username='oryx', principal='oryx/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:51824
I20260927 03:58:52.689471 24156 tablet_replica.cc:330] T d4d7d21cc8cd4039a57f2963e924aeda P 94088c272ea94c3f909bff38cd2bfbfb: stopping tablet replica
I20260927 03:58:52.689441 23899 tablet_service.cc:1514] Processing DeleteTablet for tablet d4d7d21cc8cd4039a57f2963e924aeda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:58:52 UTC) from {username='oryx', principal='oryx/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:37984
I20260927 03:58:52.689517 24156 raft_consensus.cc:2229] T d4d7d21cc8cd4039a57f2963e924aeda P 94088c272ea94c3f909bff38cd2bfbfb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:58:52.689555 24156 raft_consensus.cc:2258] T d4d7d21cc8cd4039a57f2963e924aeda P 94088c272ea94c3f909bff38cd2bfbfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:58:52.689554 24158 tablet_replica.cc:330] T d4d7d21cc8cd4039a57f2963e924aeda P 40c4cc7b32f14b0e89a6d4e89511a047: stopping tablet replica
I20260927 03:58:52.689596 24158 raft_consensus.cc:2229] T d4d7d21cc8cd4039a57f2963e924aeda P 40c4cc7b32f14b0e89a6d4e89511a047 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:58:52.689633 24158 raft_consensus.cc:2258] T d4d7d21cc8cd4039a57f2963e924aeda P 40c4cc7b32f14b0e89a6d4e89511a047 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:58:52.689793 24157 ts_tablet_manager.cc:1905] T d4d7d21cc8cd4039a57f2963e924aeda P 983a3b316b1f4809ae803c17f6c0def5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:58:52.689865 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 23694
I20260927 03:58:52.692526 24156 ts_tablet_manager.cc:1905] T d4d7d21cc8cd4039a57f2963e924aeda P 94088c272ea94c3f909bff38cd2bfbfb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:58:52.694056 24156 ts_tablet_manager.cc:1918] T d4d7d21cc8cd4039a57f2963e924aeda P 94088c272ea94c3f909bff38cd2bfbfb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:58:52.694141 24156 log.cc:1196] T d4d7d21cc8cd4039a57f2963e924aeda P 94088c272ea94c3f909bff38cd2bfbfb: Deleting WAL directory at /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/ts-2/wal/wals/d4d7d21cc8cd4039a57f2963e924aeda
I20260927 03:58:52.694360 24156 ts_tablet_manager.cc:1939] T d4d7d21cc8cd4039a57f2963e924aeda P 94088c272ea94c3f909bff38cd2bfbfb: Deleting consensus metadata
I20260927 03:58:52.694710 23607 catalog_manager.cc:4841] TS 94088c272ea94c3f909bff38cd2bfbfb (127.15.185.67:36625): tablet d4d7d21cc8cd4039a57f2963e924aeda (table test-table [id=e01a6f40b46b4fa7b81360f4b84b3ab2]) successfully deleted
I20260927 03:58:52.695319 24158 ts_tablet_manager.cc:1905] T d4d7d21cc8cd4039a57f2963e924aeda P 40c4cc7b32f14b0e89a6d4e89511a047: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:58:52.696614 24158 ts_tablet_manager.cc:1918] T d4d7d21cc8cd4039a57f2963e924aeda P 40c4cc7b32f14b0e89a6d4e89511a047: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 03:58:52.696705 24158 log.cc:1196] T d4d7d21cc8cd4039a57f2963e924aeda P 40c4cc7b32f14b0e89a6d4e89511a047: Deleting WAL directory at /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481498018407-16101-0/minicluster-data/ts-1/wal/wals/d4d7d21cc8cd4039a57f2963e924aeda
I20260927 03:58:52.696933 24158 ts_tablet_manager.cc:1939] T d4d7d21cc8cd4039a57f2963e924aeda P 40c4cc7b32f14b0e89a6d4e89511a047: Deleting consensus metadata
I20260927 03:58:52.697286 23609 catalog_manager.cc:4841] TS 40c4cc7b32f14b0e89a6d4e89511a047 (127.15.185.66:38731): tablet d4d7d21cc8cd4039a57f2963e924aeda (table test-table [id=e01a6f40b46b4fa7b81360f4b84b3ab2]) successfully deleted
W20260927 03:58:52.699281 23606 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.185.65:46859 (error 108)
I20260927 03:58:52.699518 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 23834
I20260927 03:58:52.700325 24220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:52.699641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.65:46859 (local address 127.0.0.1:45628)
0927 03:58:52.700252 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:52.700290 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.65:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":34,"threads_started":1}
W20260927 03:58:52.700472 23606 catalog_manager.cc:4584] TS 983a3b316b1f4809ae803c17f6c0def5 (127.15.185.65:46859): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d4d7d21cc8cd4039a57f2963e924aeda: Network error: Client connection negotiation failed: client connection to 127.15.185.65:46859: connect: Connection refused (error 111)
I20260927 03:58:52.706547 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 23974
I20260927 03:58:52.712013 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 23590
2026-09-27T03:58:52Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2668 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790481498018407-16101-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:58:52 dist-test-slave-glgz krb5kdc[24223](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:58:52 dist-test-slave-glgz krb5kdc[24223](info): set up 2 sockets
Sep 27 03:58:52 dist-test-slave-glgz krb5kdc[24223](info): commencing operation
krb5kdc: starting...
W20260927 03:58:53.752651 16101 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Sep 27 03:58:53 dist-test-slave-glgz krb5kdc[24223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:58:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:58:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.185.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab.
Entry for principal kudu/127.15.185.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab.
I20260927 03:58:53.870085 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790481498018407-16101-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790481498018407-16101-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790481498018407-16101-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:45571
--webserver_interface=127.15.185.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:45209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.185.126:45571
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:58:53.949747 24236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:53.950014 24236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:53.950064 24236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:53.951261 24236 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:58:53.951335 24236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:53.951359 24236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:53.951380 24236 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:58:53.951407 24236 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:58:53.952881 24236 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:45209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790481498018407-16101-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790481498018407-16101-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.185.126:45571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:45571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790481498018407-16101-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:53.953172 24236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:53.953409 24236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:53.958307 24242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:53.958308 24241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:53.958308 24244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:53.958698 24236 server_base.cc:1034] running on GCE node
I20260927 03:58:53.958945 24236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:53.959254 24236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:53.960417 24236 hybrid_clock.cc:648] HybridClock initialized: now 1790481533960409 us; error 40 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-27T03:58:53Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1254 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:58:54 dist-test-slave-glgz krb5kdc[24249](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:58:54 dist-test-slave-glgz krb5kdc[24249](info): set up 2 sockets
Sep 27 03:58:54 dist-test-slave-glgz krb5kdc[24249](info): commencing operation
krb5kdc: starting...
W20260927 03:58:55.008108 16101 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Sep 27 03:58:55 dist-test-slave-glgz krb5kdc[24249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:58:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:58:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.185.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab.
Entry for principal kudu/127.15.185.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab.
I20260927 03:58:55.129578 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:34493
--webserver_interface=127.15.185.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:35539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.185.126:34493
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:58:55.210286 24262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:55.210546 24262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:55.210583 24262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:55.211759 24262 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:58:55.211833 24262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:55.211856 24262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:55.211879 24262 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:58:55.211906 24262 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:58:55.213354 24262 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:35539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.185.126:34493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:34493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:55.213635 24262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:55.213872 24262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:55.218739 24268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:55.218732 24270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:55.218777 24262 server_base.cc:1034] running on GCE node
W20260927 03:58:55.218732 24267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:55.219219 24262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:55.219521 24262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:55.220686 24262 hybrid_clock.cc:648] HybridClock initialized: now 1790481535220673 us; error 51 us; skew 500 ppm
Sep 27 03:58:55 dist-test-slave-glgz krb5kdc[24249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481535, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:55.227716 24262 init.cc:377] Logged in from keytab as kudu/127.15.185.126@KRBTEST.COM (short username kudu)
I20260927 03:58:55.228185 24262 webserver.cc:460] Webserver started at http://127.15.185.126:33165/ using document root <none> and password file <none>
I20260927 03:58:55.228375 24262 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:55.228415 24262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:55.228510 24262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:55.229521 24262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/master-0/data/instance:
uuid: "5feec178e483456eb965044fc46e877f"
format_stamp: "Formatted at 2026-09-27 03:58:55 on dist-test-slave-glgz"
server_key: "cb57b85a8788ca9223e5dda4d72c3dbe"
server_key_iv: "7a97dbafe5a5f6273e579768e0e3a8b3"
server_key_version: "encryptionkey@0"
I20260927 03:58:55.229825 24262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/master-0/wal/instance:
uuid: "5feec178e483456eb965044fc46e877f"
format_stamp: "Formatted at 2026-09-27 03:58:55 on dist-test-slave-glgz"
server_key: "cb57b85a8788ca9223e5dda4d72c3dbe"
server_key_iv: "7a97dbafe5a5f6273e579768e0e3a8b3"
server_key_version: "encryptionkey@0"
I20260927 03:58:55.231109 24262 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 03:58:55.231792 24277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:55.231945 24262 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:58:55.232033 24262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/master-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/master-0/wal
uuid: "5feec178e483456eb965044fc46e877f"
format_stamp: "Formatted at 2026-09-27 03:58:55 on dist-test-slave-glgz"
server_key: "cb57b85a8788ca9223e5dda4d72c3dbe"
server_key_iv: "7a97dbafe5a5f6273e579768e0e3a8b3"
server_key_version: "encryptionkey@0"
I20260927 03:58:55.232087 24262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:55.244724 24262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:55.246989 24262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:55.247165 24262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:55.250679 24262 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.126:34493
I20260927 03:58:55.250778 24329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.126:34493 every 8 connection(s)
I20260927 03:58:55.251106 24262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
I20260927 03:58:55.251991 24330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:58:55.254228 24330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5feec178e483456eb965044fc46e877f: Bootstrap starting.
I20260927 03:58:55.254343 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 24262
I20260927 03:58:55.254490 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/master-0/wal/instance
I20260927 03:58:55.254698 16101 external_mini_cluster.cc:1428] Setting key e17d9270ada2e0b809cff78efd061794
I20260927 03:58:55.254953 24330 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5feec178e483456eb965044fc46e877f: Neither blocks nor log segments found. Creating new log.
I20260927 03:58:55.255362 24330 log.cc:824] T 00000000000000000000000000000000 P 5feec178e483456eb965044fc46e877f: Log is configured to *not* fsync() on all Append() calls
I20260927 03:58:55.256101 24330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5feec178e483456eb965044fc46e877f: No bootstrap required, opened a new log
I20260927 03:58:55.258400 24330 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5feec178e483456eb965044fc46e877f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5feec178e483456eb965044fc46e877f" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 34493 } }
I20260927 03:58:55.258584 24330 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5feec178e483456eb965044fc46e877f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:58:55.258632 24330 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5feec178e483456eb965044fc46e877f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5feec178e483456eb965044fc46e877f, State: Initialized, Role: FOLLOWER
I20260927 03:58:55.258769 24330 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5feec178e483456eb965044fc46e877f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5feec178e483456eb965044fc46e877f" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 34493 } }
I20260927 03:58:55.258862 24330 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5feec178e483456eb965044fc46e877f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:58:55.258904 24330 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5feec178e483456eb965044fc46e877f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:58:55.258953 24330 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5feec178e483456eb965044fc46e877f [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:58:55.259562 24330 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5feec178e483456eb965044fc46e877f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5feec178e483456eb965044fc46e877f" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 34493 } }
I20260927 03:58:55.259665 24330 leader_election.cc:302] T 00000000000000000000000000000000 P 5feec178e483456eb965044fc46e877f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5feec178e483456eb965044fc46e877f; no voters:
I20260927 03:58:55.259810 24330 leader_election.cc:288] T 00000000000000000000000000000000 P 5feec178e483456eb965044fc46e877f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:58:55.259871 24335 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5feec178e483456eb965044fc46e877f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:58:55.260064 24330 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5feec178e483456eb965044fc46e877f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:58:55.260074 24335 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5feec178e483456eb965044fc46e877f [term 1 LEADER]: Becoming Leader. State: Replica: 5feec178e483456eb965044fc46e877f, State: Running, Role: LEADER
I20260927 03:58:55.260283 24335 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5feec178e483456eb965044fc46e877f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5feec178e483456eb965044fc46e877f" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 34493 } }
I20260927 03:58:55.260697 24336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5feec178e483456eb965044fc46e877f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5feec178e483456eb965044fc46e877f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5feec178e483456eb965044fc46e877f" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 34493 } } }
I20260927 03:58:55.260774 24336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5feec178e483456eb965044fc46e877f [sys.catalog]: This master's current role is: LEADER
I20260927 03:58:55.260990 24337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5feec178e483456eb965044fc46e877f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5feec178e483456eb965044fc46e877f. Latest consensus state: current_term: 1 leader_uuid: "5feec178e483456eb965044fc46e877f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5feec178e483456eb965044fc46e877f" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 34493 } } }
I20260927 03:58:55.261283 24337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5feec178e483456eb965044fc46e877f [sys.catalog]: This master's current role is: LEADER
W20260927 03:58:55.261974 24351 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5feec178e483456eb965044fc46e877f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:58:55.262040 24351 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:58:55.262094 24344 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:58:55.262284 24344 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 03:58:55 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 03:58:55.264046 24344 catalog_manager.cc:1347] Generated new cluster ID: c2b8aef38ab34e6ab629c00a9444758a
I20260927 03:58:55.264101 24344 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:58:55.266489 24333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:55.255232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40426 (local address 127.15.185.126:34493)
0927 03:58:55.255425 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:55.255434 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:55.255474 (+ 40us) server_negotiation.cc:407] Connection header received
0927 03:58:55.256058 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:55.256067 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:55.256126 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:55.256278 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:55.256845 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:55.258318 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:55.259354 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:55.259982 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:55.260409 (+ 427us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:55.262874 (+ 2465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:55.262879 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:55.262879 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:55.262890 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:55.264842 (+ 1952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:55.265241 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:55.265244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:55.265244 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:55.265347 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:55.265546 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:55.265547 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:55.265547 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:55.265847 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:55.266009 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:55.266114 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 03:58:55.266261 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":75,"threads_started":1}
I20260927 03:58:55.277906 24344 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:58:55.278425 24344 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:58:55.280730 24344 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5feec178e483456eb965044fc46e877f: Generated new TSK 0
I20260927 03:58:55.280910 24344 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.185.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab.
Entry for principal kudu/127.15.185.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab.
I20260927 03:58:55.306286 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.65:0
--local_ip_for_outbound_sockets=127.15.185.65
--webserver_interface=127.15.185.65
--webserver_port=0
--tserver_master_addrs=127.15.185.126:34493
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab
--principal=kudu/127.15.185.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:35539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:58:55.389022 24356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:55.389235 24356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:55.389258 24356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:55.390424 24356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:55.390462 24356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:55.390488 24356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.65
I20260927 03:58:55.391932 24356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:35539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab
--principal=kudu/127.15.185.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.185.65
--webserver_port=0
--tserver_master_addrs=127.15.185.126:34493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.65
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:55.392131 24356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:55.392313 24356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 03:58:55.397279 24356 server_base.cc:1034] running on GCE node
W20260927 03:58:55.397241 24362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:55.397281 24361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:55.397260 24364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:55.397600 24356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:55.397796 24356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:55.398940 24356 hybrid_clock.cc:648] HybridClock initialized: now 1790481535398925 us; error 29 us; skew 500 ppm
Sep 27 03:58:55 dist-test-slave-glgz krb5kdc[24249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481535, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:55.401510 24356 init.cc:377] Logged in from keytab as kudu/127.15.185.65@KRBTEST.COM (short username kudu)
I20260927 03:58:55.401966 24356 webserver.cc:460] Webserver started at http://127.15.185.65:39069/ using document root <none> and password file <none>
I20260927 03:58:55.402197 24356 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:55.402244 24356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:55.402374 24356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:55.403486 24356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-0/data/instance:
uuid: "52a93449b62f4f0b8ee0b5e6c44f0559"
format_stamp: "Formatted at 2026-09-27 03:58:55 on dist-test-slave-glgz"
server_key: "2d1c70b77664dee0854f2a68a8987529"
server_key_iv: "de5ebdd491aace18e7e6de53cabca7f4"
server_key_version: "encryptionkey@0"
I20260927 03:58:55.403838 24356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-0/wal/instance:
uuid: "52a93449b62f4f0b8ee0b5e6c44f0559"
format_stamp: "Formatted at 2026-09-27 03:58:55 on dist-test-slave-glgz"
server_key: "2d1c70b77664dee0854f2a68a8987529"
server_key_iv: "de5ebdd491aace18e7e6de53cabca7f4"
server_key_version: "encryptionkey@0"
I20260927 03:58:55.405694 24356 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 03:58:55.406486 24371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:55.406663 24356 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:58:55.406776 24356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-0/wal
uuid: "52a93449b62f4f0b8ee0b5e6c44f0559"
format_stamp: "Formatted at 2026-09-27 03:58:55 on dist-test-slave-glgz"
server_key: "2d1c70b77664dee0854f2a68a8987529"
server_key_iv: "de5ebdd491aace18e7e6de53cabca7f4"
server_key_version: "encryptionkey@0"
I20260927 03:58:55.406911 24356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:55.422736 24356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:55.424686 24356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:55.424877 24356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:55.425115 24356 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:58:55.425446 24356 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:58:55.425503 24356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:55.425545 24356 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:58:55.425590 24356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:55.431231 24356 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.65:33573
I20260927 03:58:55.431308 24484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.65:33573 every 8 connection(s)
I20260927 03:58:55.431641 24356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
Sep 27 03:58:55 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481535, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.65@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 03:58:55.440909 24333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:55.432329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:60289 (local address 127.15.185.126:34493)
0927 03:58:55.432400 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:55.432402 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:55.432604 (+ 202us) server_negotiation.cc:407] Connection header received
0927 03:58:55.433102 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:55.433103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:55.433114 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:55.433178 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:55.434120 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:55.434645 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:55.435938 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:55.436210 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:55.436284 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:55.438394 (+ 2110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:55.438395 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:55.438395 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:55.438402 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:55.439777 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:55.440144 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:55.440145 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:55.440145 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:55.440191 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:55.440402 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:55.440403 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:55.440403 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:55.440506 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:55.440554 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:55.440783 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 03:58:55.440844 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 03:58:55.441138 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 24356
I20260927 03:58:55.441190 24487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:55.432322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34493 (local address 127.15.185.65:60289)
0927 03:58:55.432471 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:55.432514 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:55.432989 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:55.433232 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:55.433236 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:55.433787 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:55.433993 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:55.433998 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:55.434740 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:55.434741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:55.435850 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:55.435853 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:55.436318 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:55.436318 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:55.436423 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:55.436867 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:55.436875 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:55.438293 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:55.439859 (+ 1566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:55.439865 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:55.439866 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:55.440081 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:55.440248 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:55.440249 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:55.440249 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:55.440349 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:55.440560 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:55.440564 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:55.440675 (+ 111us) client_negotiation.cc:769] Sending connection context
0927 03:58:55.440757 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 03:58:55.440869 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":82,"threads_started":1}
I20260927 03:58:55.441262 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-0/wal/instance
I20260927 03:58:55.441489 16101 external_mini_cluster.cc:1428] Setting key 07365a9d5c4ef4caaf65004282b25f03
I20260927 03:58:55.441645 24485 heartbeater.cc:347] Connected to a master server at 127.15.185.126:34493
I20260927 03:58:55.441767 24485 heartbeater.cc:464] Registering TS with master...
I20260927 03:58:55.441954 24485 heartbeater.cc:511] Master 127.15.185.126:34493 requested a full tablet report, sending...
I20260927 03:58:55.442492 24294 ts_manager.cc:194] Registered new tserver with Master: 52a93449b62f4f0b8ee0b5e6c44f0559 (127.15.185.65:33573)
I20260927 03:58:55.443107 24294 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.185.65@KRBTEST.COM'} at 127.15.185.65:60289
WARNING: no policy specified for kudu/127.15.185.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab.
Entry for principal kudu/127.15.185.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab.
I20260927 03:58:55.465631 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.66:0
--local_ip_for_outbound_sockets=127.15.185.66
--webserver_interface=127.15.185.66
--webserver_port=0
--tserver_master_addrs=127.15.185.126:34493
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab
--principal=kudu/127.15.185.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:35539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:58:55.545310 24490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:55.545527 24490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:55.545549 24490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:55.546624 24490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:55.546659 24490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:55.546684 24490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.66
I20260927 03:58:55.548060 24490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:35539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab
--principal=kudu/127.15.185.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.185.66
--webserver_port=0
--tserver_master_addrs=127.15.185.126:34493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.66
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:55.548241 24490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:55.548411 24490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:55.553340 24498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:55.553347 24496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:55.553443 24490 server_base.cc:1034] running on GCE node
W20260927 03:58:55.553344 24495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:55.553851 24490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:55.554103 24490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:55.555274 24490 hybrid_clock.cc:648] HybridClock initialized: now 1790481535555263 us; error 30 us; skew 500 ppm
Sep 27 03:58:55 dist-test-slave-glgz krb5kdc[24249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481535, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:55.557971 24490 init.cc:377] Logged in from keytab as kudu/127.15.185.66@KRBTEST.COM (short username kudu)
I20260927 03:58:55.558480 24490 webserver.cc:460] Webserver started at http://127.15.185.66:46599/ using document root <none> and password file <none>
I20260927 03:58:55.558717 24490 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:55.558784 24490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:55.558934 24490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:55.560220 24490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-1/data/instance:
uuid: "465e8c693abf48f98040dceb86f1368f"
format_stamp: "Formatted at 2026-09-27 03:58:55 on dist-test-slave-glgz"
server_key: "e3d65495204e7ed13388dc21c9811431"
server_key_iv: "4dfed1d11771c9bb1a7fc35ed9ef1471"
server_key_version: "encryptionkey@0"
I20260927 03:58:55.560662 24490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-1/wal/instance:
uuid: "465e8c693abf48f98040dceb86f1368f"
format_stamp: "Formatted at 2026-09-27 03:58:55 on dist-test-slave-glgz"
server_key: "e3d65495204e7ed13388dc21c9811431"
server_key_iv: "4dfed1d11771c9bb1a7fc35ed9ef1471"
server_key_version: "encryptionkey@0"
I20260927 03:58:55.562175 24490 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:58:55.562915 24505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:55.563125 24490 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:58:55.563221 24490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-1/wal
uuid: "465e8c693abf48f98040dceb86f1368f"
format_stamp: "Formatted at 2026-09-27 03:58:55 on dist-test-slave-glgz"
server_key: "e3d65495204e7ed13388dc21c9811431"
server_key_iv: "4dfed1d11771c9bb1a7fc35ed9ef1471"
server_key_version: "encryptionkey@0"
I20260927 03:58:55.563299 24490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:55.593320 24490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:55.595764 24490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:55.595933 24490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:55.596175 24490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:58:55.596493 24490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:58:55.596551 24490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:55.596592 24490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:58:55.596621 24490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:55.602803 24490 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.66:45647
I20260927 03:58:55.602852 24618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.66:45647 every 8 connection(s)
I20260927 03:58:55.603240 24490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
Sep 27 03:58:55 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481535, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.66@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 03:58:55.610478 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 24490
I20260927 03:58:55.610608 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-1/wal/instance
I20260927 03:58:55.610836 16101 external_mini_cluster.cc:1428] Setting key c9fc7ebf0a6454fb19a2f60be3ab3e1b
I20260927 03:58:55.612862 24333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:55.604033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:36215 (local address 127.15.185.126:34493)
0927 03:58:55.604098 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:55.604100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:55.604325 (+ 225us) server_negotiation.cc:407] Connection header received
0927 03:58:55.604801 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:55.604802 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:55.604812 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:55.604869 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:55.605796 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:55.606298 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:55.607303 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:55.607503 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:55.607565 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:55.609774 (+ 2209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:55.609776 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:55.609776 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:55.609783 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:55.611340 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:55.611824 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:55.611824 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:55.611825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:55.611876 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:55.612163 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:55.612163 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:55.612163 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:55.612264 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:55.612320 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:55.612701 (+ 381us) server_negotiation.cc:299] Negotiation successful
0927 03:58:55.612773 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 03:58:55.613163 24621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:55.604030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34493 (local address 127.15.185.66:36215)
0927 03:58:55.604188 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:55.604226 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:55.604705 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:55.604923 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:55.604926 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:55.605497 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:55.605704 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:55.605709 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:55.606384 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:55.606385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:55.607212 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:55.607214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:55.607583 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:55.607583 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:55.607682 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:55.608269 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:55.608278 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:55.609672 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:55.611407 (+ 1735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:55.611415 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:55.611418 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:55.611740 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:55.611933 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:55.611934 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:55.611934 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:55.612099 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:55.612329 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:55.612335 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:55.612526 (+ 191us) client_negotiation.cc:769] Sending connection context
0927 03:58:55.612650 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 03:58:55.612849 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":93,"threads_started":1}
I20260927 03:58:55.613683 24619 heartbeater.cc:347] Connected to a master server at 127.15.185.126:34493
I20260927 03:58:55.613770 24619 heartbeater.cc:464] Registering TS with master...
I20260927 03:58:55.613983 24619 heartbeater.cc:511] Master 127.15.185.126:34493 requested a full tablet report, sending...
I20260927 03:58:55.614414 24294 ts_manager.cc:194] Registered new tserver with Master: 465e8c693abf48f98040dceb86f1368f (127.15.185.66:45647)
I20260927 03:58:55.614794 24294 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.185.66@KRBTEST.COM'} at 127.15.185.66:36215
WARNING: no policy specified for kudu/127.15.185.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab.
Entry for principal kudu/127.15.185.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab.
I20260927 03:58:55.638111 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.67:0
--local_ip_for_outbound_sockets=127.15.185.67
--webserver_interface=127.15.185.67
--webserver_port=0
--tserver_master_addrs=127.15.185.126:34493
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab
--principal=kudu/127.15.185.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:35539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:58:55.720192 24624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:55.720405 24624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:55.720428 24624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:55.721508 24624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:55.721542 24624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:55.721570 24624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.67
I20260927 03:58:55.723014 24624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:35539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab
--principal=kudu/127.15.185.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.185.67
--webserver_port=0
--tserver_master_addrs=127.15.185.126:34493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.67
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:55.723310 24624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:55.723551 24624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:55.728595 24630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:55.728583 24629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:55.728583 24632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:55.728909 24624 server_base.cc:1034] running on GCE node
I20260927 03:58:55.729115 24624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:55.729360 24624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:55.730521 24624 hybrid_clock.cc:648] HybridClock initialized: now 1790481535730520 us; error 40 us; skew 500 ppm
Sep 27 03:58:55 dist-test-slave-glgz krb5kdc[24249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481535, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:55.732949 24624 init.cc:377] Logged in from keytab as kudu/127.15.185.67@KRBTEST.COM (short username kudu)
I20260927 03:58:55.733412 24624 webserver.cc:460] Webserver started at http://127.15.185.67:36835/ using document root <none> and password file <none>
I20260927 03:58:55.733623 24624 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:55.733664 24624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:55.733757 24624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:58:55.734874 24624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-2/data/instance:
uuid: "b1be0aae14ce45f29ac8e3b116fcdc47"
format_stamp: "Formatted at 2026-09-27 03:58:55 on dist-test-slave-glgz"
server_key: "b213e9162ba009ce029be7c0d846ec29"
server_key_iv: "9e969e0ecd4fd128c25a8520b76ba1a9"
server_key_version: "encryptionkey@0"
I20260927 03:58:55.735236 24624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-2/wal/instance:
uuid: "b1be0aae14ce45f29ac8e3b116fcdc47"
format_stamp: "Formatted at 2026-09-27 03:58:55 on dist-test-slave-glgz"
server_key: "b213e9162ba009ce029be7c0d846ec29"
server_key_iv: "9e969e0ecd4fd128c25a8520b76ba1a9"
server_key_version: "encryptionkey@0"
I20260927 03:58:55.736634 24624 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:58:55.737355 24639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:55.737594 24624 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:58:55.737694 24624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-2/wal
uuid: "b1be0aae14ce45f29ac8e3b116fcdc47"
format_stamp: "Formatted at 2026-09-27 03:58:55 on dist-test-slave-glgz"
server_key: "b213e9162ba009ce029be7c0d846ec29"
server_key_iv: "9e969e0ecd4fd128c25a8520b76ba1a9"
server_key_version: "encryptionkey@0"
I20260927 03:58:55.737756 24624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:55.766229 24624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:55.768543 24624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:55.768664 24624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:55.768865 24624 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:58:55.769140 24624 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:58:55.769171 24624 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:55.769194 24624 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:58:55.769207 24624 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:55.774772 24624 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.67:40125
I20260927 03:58:55.774837 24752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.67:40125 every 8 connection(s)
I20260927 03:58:55.775223 24624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
Sep 27 03:58:55 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481535, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.67@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 03:58:55.783533 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 24624
I20260927 03:58:55.783665 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-2/wal/instance
I20260927 03:58:55.783933 16101 external_mini_cluster.cc:1428] Setting key 9839c33c018a23e428b1cdeaf26cc603
I20260927 03:58:55.784633 24333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:55.775982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:60629 (local address 127.15.185.126:34493)
0927 03:58:55.776026 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:55.776028 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:55.776254 (+ 226us) server_negotiation.cc:407] Connection header received
0927 03:58:55.776793 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:55.776794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:55.776807 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:55.776871 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:55.777838 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:55.778346 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:55.779401 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:55.779602 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:55.779686 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:55.781942 (+ 2256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:55.781944 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:55.781944 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:55.781952 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:55.783282 (+ 1330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:55.783676 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:55.783677 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:55.783678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:55.783764 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:55.783995 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:55.783996 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:55.783997 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:55.784120 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:55.784198 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:55.784459 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 03:58:55.784536 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 03:58:55.784854 24755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:55.775959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34493 (local address 127.15.185.67:60629)
0927 03:58:55.776135 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:55.776170 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:55.776681 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:55.776955 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:55.776959 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:55.777527 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:55.777719 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:55.777723 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:55.778449 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:55.778450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:55.779317 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:55.779319 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:55.779693 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:55.779694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:55.779821 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:55.780312 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:55.780320 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:55.781846 (+ 1526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:58:55.783370 (+ 1524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:55.783377 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:55.783379 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:55.783598 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:55.783818 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:58:55.783819 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:58:55.783819 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:58:55.783926 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:58:55.784186 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:58:55.784192 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:58:55.784331 (+ 139us) client_negotiation.cc:769] Sending connection context
0927 03:58:55.784440 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 03:58:55.784606 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":84,"threads_started":1}
I20260927 03:58:55.785343 24753 heartbeater.cc:347] Connected to a master server at 127.15.185.126:34493
I20260927 03:58:55.785447 24753 heartbeater.cc:464] Registering TS with master...
I20260927 03:58:55.785636 24753 heartbeater.cc:511] Master 127.15.185.126:34493 requested a full tablet report, sending...
I20260927 03:58:55.786006 24294 ts_manager.cc:194] Registered new tserver with Master: b1be0aae14ce45f29ac8e3b116fcdc47 (127.15.185.67:40125)
I20260927 03:58:55.786470 24294 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.185.67@KRBTEST.COM'} at 127.15.185.67:60629
I20260927 03:58:55.795818 16101 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.15.185.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.185.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.185.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/krb5kdc/oryx_127.15.185.65.keytab.
Entry for principal oryx/127.15.185.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/krb5kdc/oryx_127.15.185.65.keytab.
WARNING: no policy specified for oryx/127.15.185.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.185.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.185.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/krb5kdc/oryx_127.15.185.66.keytab.
Entry for principal oryx/127.15.185.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/krb5kdc/oryx_127.15.185.66.keytab.
WARNING: no policy specified for oryx/127.15.185.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.185.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.185.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/krb5kdc/oryx_127.15.185.67.keytab.
Entry for principal oryx/127.15.185.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/krb5kdc/oryx_127.15.185.67.keytab.
I20260927 03:58:55.870015 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 24356
I20260927 03:58:55.875263 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 24490
I20260927 03:58:55.880124 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 24624
I20260927 03:58:55.884972 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 24262
I20260927 03:58:55.890254 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:34493
--webserver_interface=127.15.185.126
--webserver_port=33165
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:35539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.185.126:34493
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:58:55.970920 24762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:55.971164 24762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:55.971189 24762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:55.972429 24762 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:58:55.972489 24762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:55.972514 24762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:55.972534 24762 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:58:55.972546 24762 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:58:55.973989 24762 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:35539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.185.126:34493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:34493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.185.126
--webserver_port=33165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:55.974195 24762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:55.974388 24762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:55.979275 24767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:55.979264 24768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:55.979295 24762 server_base.cc:1034] running on GCE node
W20260927 03:58:55.979261 24770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:55.979655 24762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:55.979949 24762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:55.981112 24762 hybrid_clock.cc:648] HybridClock initialized: now 1790481535981098 us; error 27 us; skew 500 ppm
Sep 27 03:58:55 dist-test-slave-glgz krb5kdc[24249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481535, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:55.983706 24762 init.cc:377] Logged in from keytab as kudu/127.15.185.126@KRBTEST.COM (short username kudu)
I20260927 03:58:55.984086 24762 webserver.cc:460] Webserver started at http://127.15.185.126:33165/ using document root <none> and password file <none>
I20260927 03:58:55.984265 24762 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:55.984315 24762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:55.985587 24762 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:58:55.986202 24777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:55.986387 24762 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:58:55.986485 24762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/master-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/master-0/wal
uuid: "5feec178e483456eb965044fc46e877f"
format_stamp: "Formatted at 2026-09-27 03:58:55 on dist-test-slave-glgz"
server_key: "cb57b85a8788ca9223e5dda4d72c3dbe"
server_key_iv: "7a97dbafe5a5f6273e579768e0e3a8b3"
server_key_version: "encryptionkey@0"
I20260927 03:58:55.986817 24762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:56.006265 24762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:56.008244 24762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:56.008405 24762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:56.012416 24762 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.126:34493
I20260927 03:58:56.012503 24829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.126:34493 every 8 connection(s)
I20260927 03:58:56.012931 24762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
I20260927 03:58:56.014422 24830 sys_catalog.cc:263] Verifying existing consensus state
I20260927 03:58:56.015040 24830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5feec178e483456eb965044fc46e877f: Bootstrap starting.
I20260927 03:58:56.015236 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 24762
I20260927 03:58:56.015550 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.65:33573
--local_ip_for_outbound_sockets=127.15.185.65
--tserver_master_addrs=127.15.185.126:34493
--webserver_port=39069
--webserver_interface=127.15.185.65
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab
--principal=kudu/127.15.185.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:35539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.15.185.65
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/krb5kdc/oryx_127.15.185.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 03:58:56.023674 24830 log.cc:824] T 00000000000000000000000000000000 P 5feec178e483456eb965044fc46e877f: Log is configured to *not* fsync() on all Append() calls
I20260927 03:58:56.025489 24830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5feec178e483456eb965044fc46e877f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 03:58:56.025812 24830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5feec178e483456eb965044fc46e877f: Bootstrap complete.
I20260927 03:58:56.028136 24830 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5feec178e483456eb965044fc46e877f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5feec178e483456eb965044fc46e877f" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 34493 } }
I20260927 03:58:56.028379 24830 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5feec178e483456eb965044fc46e877f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5feec178e483456eb965044fc46e877f, State: Initialized, Role: FOLLOWER
I20260927 03:58:56.028594 24830 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5feec178e483456eb965044fc46e877f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5feec178e483456eb965044fc46e877f" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 34493 } }
I20260927 03:58:56.028656 24830 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5feec178e483456eb965044fc46e877f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:58:56.028703 24830 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5feec178e483456eb965044fc46e877f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:58:56.028751 24830 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5feec178e483456eb965044fc46e877f [term 1 FOLLOWER]: Advancing to term 2
I20260927 03:58:56.029712 24830 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5feec178e483456eb965044fc46e877f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5feec178e483456eb965044fc46e877f" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 34493 } }
I20260927 03:58:56.029794 24830 leader_election.cc:302] T 00000000000000000000000000000000 P 5feec178e483456eb965044fc46e877f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5feec178e483456eb965044fc46e877f; no voters:
I20260927 03:58:56.030001 24830 leader_election.cc:288] T 00000000000000000000000000000000 P 5feec178e483456eb965044fc46e877f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 03:58:56.030043 24834 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5feec178e483456eb965044fc46e877f [term 2 FOLLOWER]: Leader election won for term 2
I20260927 03:58:56.030191 24834 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5feec178e483456eb965044fc46e877f [term 2 LEADER]: Becoming Leader. State: Replica: 5feec178e483456eb965044fc46e877f, State: Running, Role: LEADER
I20260927 03:58:56.030268 24830 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5feec178e483456eb965044fc46e877f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:58:56.030334 24834 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5feec178e483456eb965044fc46e877f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5feec178e483456eb965044fc46e877f" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 34493 } }
I20260927 03:58:56.030628 24834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5feec178e483456eb965044fc46e877f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5feec178e483456eb965044fc46e877f. Latest consensus state: current_term: 2 leader_uuid: "5feec178e483456eb965044fc46e877f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5feec178e483456eb965044fc46e877f" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 34493 } } }
I20260927 03:58:56.030633 24836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5feec178e483456eb965044fc46e877f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5feec178e483456eb965044fc46e877f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5feec178e483456eb965044fc46e877f" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 34493 } } }
I20260927 03:58:56.030784 24834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5feec178e483456eb965044fc46e877f [sys.catalog]: This master's current role is: LEADER
I20260927 03:58:56.030794 24836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5feec178e483456eb965044fc46e877f [sys.catalog]: This master's current role is: LEADER
I20260927 03:58:56.031014 24841 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:58:56.031590 24841 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:58:56.031770 24841 catalog_manager.cc:1259] Loaded cluster ID: c2b8aef38ab34e6ab629c00a9444758a
I20260927 03:58:56.031944 24841 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:58:56.033098 24841 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:58:56.033248 24841 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5feec178e483456eb965044fc46e877f: Loaded TSK: 0
I20260927 03:58:56.033442 24841 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 03:58:56.098670 24832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:56.098905 24832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:56.098932 24832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:56.100282 24832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:56.100328 24832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:56.100358 24832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.65
I20260927 03:58:56.101778 24832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:35539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/krb5kdc/oryx_127.15.185.65.keytab
--principal=oryx/127.15.185.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.65:33573
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.185.65
--webserver_port=39069
--tserver_master_addrs=127.15.185.126:34493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.65
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:56.101981 24832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:56.102165 24832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:56.107321 24857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:56.107304 24856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:56.107460 24832 server_base.cc:1034] running on GCE node
W20260927 03:58:56.107308 24859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:56.107788 24832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:56.108088 24832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:56.109272 24832 hybrid_clock.cc:648] HybridClock initialized: now 1790481536109248 us; error 39 us; skew 500 ppm
Sep 27 03:58:56 dist-test-slave-glgz krb5kdc[24249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481536, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.185.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:56.112340 24832 init.cc:377] Logged in from keytab as oryx/127.15.185.65@KRBTEST.COM (short username oryx)
I20260927 03:58:56.112867 24832 webserver.cc:460] Webserver started at http://127.15.185.65:39069/ using document root <none> and password file <none>
I20260927 03:58:56.113154 24832 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:56.113237 24832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:56.114666 24832 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:58:56.115370 24866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:56.115530 24832 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:58:56.115619 24832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-0/wal
uuid: "52a93449b62f4f0b8ee0b5e6c44f0559"
format_stamp: "Formatted at 2026-09-27 03:58:55 on dist-test-slave-glgz"
server_key: "2d1c70b77664dee0854f2a68a8987529"
server_key_iv: "de5ebdd491aace18e7e6de53cabca7f4"
server_key_version: "encryptionkey@0"
I20260927 03:58:56.115938 24832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:56.128120 24832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:56.130431 24832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:56.130764 24832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:56.131114 24832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:58:56.131495 24832 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:58:56.131552 24832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:56.131596 24832 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:58:56.131644 24832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:56.138177 24832 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.65:33573
I20260927 03:58:56.138221 24979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.65:33573 every 8 connection(s)
I20260927 03:58:56.138615 24832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
I20260927 03:58:56.140935 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 24832
I20260927 03:58:56.141279 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.66:45647
--local_ip_for_outbound_sockets=127.15.185.66
--tserver_master_addrs=127.15.185.126:34493
--webserver_port=46599
--webserver_interface=127.15.185.66
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab
--principal=kudu/127.15.185.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:35539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.15.185.66
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/krb5kdc/oryx_127.15.185.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 03:58:56 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.65@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:58:56 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.65@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:58:56.149683 24982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:56.139490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34493 (local address 127.15.185.65:49715)
0927 03:58:56.139666 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:56.139716 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:56.140188 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:56.140648 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:56.140652 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:56.141525 (+ 873us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:56.141814 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:56.141821 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:56.143495 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:56.143496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:56.144845 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:56.144848 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:56.145661 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:56.145663 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:56.145817 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:56.146448 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:56.146461 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:56.149424 (+ 2963us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.185.126:34493: Server oryx/127.15.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":92,"threads_started":1}
W20260927 03:58:56.149752 24983 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:58:56.139607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:49715 (local address 127.15.185.126:34493)
0927 03:58:56.139787 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:56.139796 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:56.139839 (+ 43us) server_negotiation.cc:407] Connection header received
0927 03:58:56.140404 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:56.140412 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:56.140468 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:56.140610 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:56.141945 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:56.143348 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:56.144948 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:56.145562 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:56.145665 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:56.149570 (+ 3905us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.185.65:49715: BlockingRecv error: recv got EOF from 127.15.185.65:49715 (error 108)
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":78,"threads_started":1}
W20260927 03:58:56.150102 24980 heartbeater.cc:644] Failed to heartbeat to 127.15.185.126:34493 (0 consecutive failures): Not authorized: Failed to ping master at 127.15.185.126:34493: Client connection negotiation failed: client connection to 127.15.185.126:34493: Server oryx/127.15.185.126@KRBTEST.COM not found in Kerberos database
Sep 27 03:58:56 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.65@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:58:56 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.65@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:58:56.155941 24982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:56.150398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34493 (local address 127.15.185.65:58993)
0927 03:58:56.150614 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:56.150626 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:56.150667 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:56.150879 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:56.150880 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:56.151032 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:56.151100 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:56.151101 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:56.152107 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:56.152108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:56.152850 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:56.152853 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:56.153325 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:56.153326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:56.153398 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:56.153880 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:56.153885 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:56.155867 (+ 1982us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.185.126:34493: Server oryx/127.15.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198}
W20260927 03:58:56.156195 24985 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:58:56.150469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:58993 (local address 127.15.185.126:34493)
0927 03:58:56.150653 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:56.150656 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:56.150667 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:58:56.150720 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:56.150721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:56.150733 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:56.150847 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:56.151188 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:56.152011 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:56.152937 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:56.153249 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:56.153318 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:56.155965 (+ 2647us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.185.65:58993: BlockingRecv error: recv got EOF from 127.15.185.65:58993 (error 108)
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":109,"threads_started":1}
Sep 27 03:58:56 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.65@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:58:56 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.65@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:58:56.161309 24982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:56.156251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34493 (local address 127.15.185.65:54455)
0927 03:58:56.156373 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:56.156383 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:56.156416 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:56.156546 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:56.156547 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:56.156676 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:56.156707 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:56.156708 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:56.157606 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:56.157606 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:56.158321 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:56.158323 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:56.158809 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:56.158810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:56.158859 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:56.159368 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:56.159373 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:56.161242 (+ 1869us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.185.126:34493: Server oryx/127.15.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106}
W20260927 03:58:56.161391 24985 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:58:56.156327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:54455 (local address 127.15.185.126:34493)
0927 03:58:56.156413 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:56.156415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:56.156425 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:58:56.156476 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:56.156477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:56.156487 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:56.156544 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:56.156772 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:56.157521 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:56.158412 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:56.158726 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:56.158796 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:56.161334 (+ 2538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.185.65:54455: BlockingRecv error: recv got EOF from 127.15.185.65:54455 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260927 03:58:56.161734 24980 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 03:58:56.229729 24984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:56.229943 24984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:56.229974 24984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:56.231264 24984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:56.231305 24984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:56.231333 24984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.66
I20260927 03:58:56.232771 24984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:35539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/krb5kdc/oryx_127.15.185.66.keytab
--principal=oryx/127.15.185.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.66:45647
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.185.66
--webserver_port=46599
--tserver_master_addrs=127.15.185.126:34493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.66
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:56.233017 24984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:56.233235 24984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 03:58:56.238250 24984 server_base.cc:1034] running on GCE node
W20260927 03:58:56.238219 24990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:56.238199 24993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:56.238268 24991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:56.238752 24984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:56.239005 24984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:56.240180 24984 hybrid_clock.cc:648] HybridClock initialized: now 1790481536240162 us; error 38 us; skew 500 ppm
Sep 27 03:58:56 dist-test-slave-glgz krb5kdc[24249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481536, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.185.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:56.242626 24984 init.cc:377] Logged in from keytab as oryx/127.15.185.66@KRBTEST.COM (short username oryx)
I20260927 03:58:56.243124 24984 webserver.cc:460] Webserver started at http://127.15.185.66:46599/ using document root <none> and password file <none>
I20260927 03:58:56.243352 24984 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:56.243431 24984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:56.244778 24984 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:58:56.245357 25000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:56.245513 24984 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:58:56.245610 24984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-1/wal
uuid: "465e8c693abf48f98040dceb86f1368f"
format_stamp: "Formatted at 2026-09-27 03:58:55 on dist-test-slave-glgz"
server_key: "e3d65495204e7ed13388dc21c9811431"
server_key_iv: "4dfed1d11771c9bb1a7fc35ed9ef1471"
server_key_version: "encryptionkey@0"
I20260927 03:58:56.245985 24984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:56.255837 24984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:56.258062 24984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:56.258224 24984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:56.258446 24984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:58:56.258747 24984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:58:56.258806 24984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:56.258848 24984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:58:56.258876 24984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:56.264848 24984 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.66:45647
I20260927 03:58:56.264891 25113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.66:45647 every 8 connection(s)
I20260927 03:58:56.265332 24984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
I20260927 03:58:56.266700 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 24984
I20260927 03:58:56.267068 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.67:40125
--local_ip_for_outbound_sockets=127.15.185.67
--tserver_master_addrs=127.15.185.126:34493
--webserver_port=36835
--webserver_interface=127.15.185.67
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab
--principal=kudu/127.15.185.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:35539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.15.185.67
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/krb5kdc/oryx_127.15.185.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 03:58:56 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.66@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:58:56 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.66@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
W20260927 03:58:56.274891 24985 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:58:56.266414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:44751 (local address 127.15.185.126:34493)
0927 03:58:56.266709 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:56.266711 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:56.266842 (+ 131us) server_negotiation.cc:407] Connection header received
0927 03:58:56.267664 (+ 822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:56.267665 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:56.267678 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:56.267731 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:56.268919 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:56.269697 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:56.271334 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:56.271525 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:56.271577 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:56.274832 (+ 3255us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.185.66:44751: BlockingRecv error: recv got EOF from 127.15.185.66:44751 (error 108)
Metrics: {"server-negotiator.queue_time_us":272}
I20260927 03:58:56.274981 25116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:56.266451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34493 (local address 127.15.185.66:44751)
0927 03:58:56.266685 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:56.266736 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:56.267536 (+ 800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:56.267803 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:56.267808 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:56.268609 (+ 801us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:56.268814 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:56.268819 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:56.269788 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:56.269789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:56.271239 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:56.271242 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:56.271608 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:56.271609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:56.271777 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:56.272298 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:56.272310 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:56.274769 (+ 2459us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.185.126:34493: Server oryx/127.15.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":79,"threads_started":1}
W20260927 03:58:56.275206 25114 heartbeater.cc:644] Failed to heartbeat to 127.15.185.126:34493 (0 consecutive failures): Not authorized: Failed to ping master at 127.15.185.126:34493: Client connection negotiation failed: client connection to 127.15.185.126:34493: Server oryx/127.15.185.126@KRBTEST.COM not found in Kerberos database
Sep 27 03:58:56 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.66@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:58:56 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.66@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:58:56.279513 25116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:56.275423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34493 (local address 127.15.185.66:39715)
0927 03:58:56.275473 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:56.275484 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:56.275525 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:56.275676 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:56.275677 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:56.275800 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:56.275840 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:56.275842 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:56.276441 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:56.276443 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:56.276939 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:56.276940 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:56.277304 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:56.277305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:56.277368 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:56.277785 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:56.277790 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:56.279446 (+ 1656us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.185.126:34493: Server oryx/127.15.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 03:58:56.279619 24985 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:58:56.275495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:39715 (local address 127.15.185.126:34493)
0927 03:58:56.275552 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:56.275554 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:56.275562 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:58:56.275583 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:56.275583 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:56.275593 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:56.275659 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:56.275920 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:56.276365 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:56.277034 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:56.277221 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:56.277303 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:56.279532 (+ 2229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.185.66:39715: BlockingRecv error: recv got EOF from 127.15.185.66:39715 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 27 03:58:56 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.66@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:58:56 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.66@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:58:56.283934 25116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:56.279875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34493 (local address 127.15.185.66:50833)
0927 03:58:56.279923 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:56.279932 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:56.280013 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:56.280156 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:56.280157 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:56.280272 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:56.280295 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:56.280296 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:56.280945 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:56.280946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:56.281422 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:56.281423 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:56.281785 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:56.281785 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:56.281836 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:56.282227 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:56.282232 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:56.283858 (+ 1626us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.185.126:34493: Server oryx/127.15.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260927 03:58:56.284060 25114 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 03:58:56.283980 24985 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:58:56.279942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:50833 (local address 127.15.185.126:34493)
0927 03:58:56.279996 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:56.279998 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:56.280008 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:58:56.280071 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:56.280072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:56.280081 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:56.280151 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:56.280380 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:56.280823 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:56.281521 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:56.281703 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:56.281784 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:56.283922 (+ 2138us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.185.66:50833: BlockingRecv error: recv got EOF from 127.15.185.66:50833 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260927 03:58:56.351739 25117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:58:56.352041 25117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:58:56.352084 25117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:58:56.353268 25117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:58:56.353334 25117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:58:56.353377 25117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.67
I20260927 03:58:56.354811 25117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:35539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/krb5kdc/oryx_127.15.185.67.keytab
--principal=oryx/127.15.185.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.67:40125
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.185.67
--webserver_port=36835
--tserver_master_addrs=127.15.185.126:34493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.67
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:58:56.355095 25117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:58:56.355337 25117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:58:56.360337 25123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:58:56.360404 25122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:56.360374 25117 server_base.cc:1034] running on GCE node
W20260927 03:58:56.360486 25125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:58:56.360823 25117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:58:56.361029 25117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:58:56.362161 25117 hybrid_clock.cc:648] HybridClock initialized: now 1790481536362141 us; error 37 us; skew 500 ppm
Sep 27 03:58:56 dist-test-slave-glgz krb5kdc[24249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481536, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.185.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:58:56.364500 25117 init.cc:377] Logged in from keytab as oryx/127.15.185.67@KRBTEST.COM (short username oryx)
I20260927 03:58:56.364934 25117 webserver.cc:460] Webserver started at http://127.15.185.67:36835/ using document root <none> and password file <none>
I20260927 03:58:56.365156 25117 fs_manager.cc:359] Metadata directory not provided
I20260927 03:58:56.365235 25117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:58:56.366703 25117 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:58:56.367365 25132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:56.367547 25117 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 03:58:56.367638 25117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-2/wal
uuid: "b1be0aae14ce45f29ac8e3b116fcdc47"
format_stamp: "Formatted at 2026-09-27 03:58:55 on dist-test-slave-glgz"
server_key: "b213e9162ba009ce029be7c0d846ec29"
server_key_iv: "9e969e0ecd4fd128c25a8520b76ba1a9"
server_key_version: "encryptionkey@0"
I20260927 03:58:56.367975 25117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:58:56.375439 25117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:58:56.377128 25117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:58:56.377274 25117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:58:56.377471 25117 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:58:56.377748 25117 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:58:56.377799 25117 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:56.377838 25117 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:58:56.377864 25117 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:58:56.383601 25117 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.67:40125
I20260927 03:58:56.383638 25245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.67:40125 every 8 connection(s)
I20260927 03:58:56.383978 25117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
Sep 27 03:58:56 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.67@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:58:56 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.67@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
W20260927 03:58:56.391378 24985 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:58:56.384656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:49717 (local address 127.15.185.126:34493)
0927 03:58:56.384700 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:56.384703 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:56.384922 (+ 219us) server_negotiation.cc:407] Connection header received
0927 03:58:56.385373 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:56.385374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:56.385383 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:56.385442 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:56.386347 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:56.386826 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:56.388022 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:56.388254 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:56.388320 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:56.391334 (+ 3014us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.185.67:49717: BlockingRecv error: recv got EOF from 127.15.185.67:49717 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 03:58:56.391402 25248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:56.384641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34493 (local address 127.15.185.67:49717)
0927 03:58:56.384792 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:56.384832 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:56.385295 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:56.385529 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:56.385533 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:56.386083 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:56.386273 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:56.386278 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:56.387170 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:56.387171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:56.387942 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:56.387944 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:56.388336 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:56.388336 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:56.388428 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:56.388969 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:56.388979 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:56.391255 (+ 2276us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.185.126:34493: Server oryx/127.15.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":87,"threads_started":1}
W20260927 03:58:56.391644 25246 heartbeater.cc:644] Failed to heartbeat to 127.15.185.126:34493 (0 consecutive failures): Not authorized: Failed to ping master at 127.15.185.126:34493: Client connection negotiation failed: client connection to 127.15.185.126:34493: Server oryx/127.15.185.126@KRBTEST.COM not found in Kerberos database
I20260927 03:58:56.392596 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 25117
Sep 27 03:58:56 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.67@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:58:56 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.67@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:58:56.396837 25248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:56.391878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34493 (local address 127.15.185.67:40445)
0927 03:58:56.391931 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:56.391944 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:56.391988 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:56.392224 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:56.392225 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:56.392329 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:56.392372 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:56.392373 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:56.393123 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:56.393124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:56.393992 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:56.393994 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:56.394421 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:56.394422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:56.394468 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:56.394912 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:56.394918 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:56.396756 (+ 1838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.185.126:34493: Server oryx/127.15.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 03:58:56.396878 24985 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:58:56.392067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:40445 (local address 127.15.185.126:34493)
0927 03:58:56.392120 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:56.392121 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:56.392129 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:58:56.392150 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:56.392151 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:56.392159 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:56.392211 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:56.392432 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:56.393047 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:56.394081 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:56.394351 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:56.394417 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:56.396817 (+ 2400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.185.67:40445: BlockingRecv error: recv got EOF from 127.15.185.67:40445 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 27 03:58:56 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.67@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:58:56 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.67@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:58:56.400851 24983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:56.393025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40430 (local address 127.15.185.126:34493)
0927 03:58:56.393072 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:56.393074 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:56.393238 (+ 164us) server_negotiation.cc:407] Connection header received
0927 03:58:56.393296 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:56.393297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:56.393308 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:56.393348 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:56.394035 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:56.394694 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:56.395763 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:56.395978 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:56.396075 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:56.397392 (+ 1317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:58:56.397397 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:58:56.397397 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:58:56.397410 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:58:56.399023 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:56.399463 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:56.399469 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:56.399469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:56.399570 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:58:56.400161 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:58:56.400162 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:58:56.400162 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:58:56.400384 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:58:56.400539 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 03:58:56.400653 (+ 114us) server_negotiation.cc:299] Negotiation successful
0927 03:58:56.400769 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 03:58:56.401062 25248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:56.397118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34493 (local address 127.15.185.67:43165)
0927 03:58:56.397155 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:56.397164 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:56.397197 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:56.397377 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:56.397378 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:56.397502 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:56.397524 (+ 22us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:56.397525 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:56.398098 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:56.398099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:56.398610 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:56.398611 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:56.398957 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:56.398958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:56.399015 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:56.399419 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:56.399423 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:56.401005 (+ 1582us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.185.126:34493: Server oryx/127.15.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":24}
W20260927 03:58:56.401110 24985 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:58:56.397227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:43165 (local address 127.15.185.126:34493)
0927 03:58:56.397284 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:56.397285 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:56.397291 (+ 6us) server_negotiation.cc:407] Connection header received
0927 03:58:56.397305 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:56.397306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:56.397312 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:56.397373 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:56.397594 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:56.398022 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:56.398691 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:56.398878 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:56.398947 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:56.401072 (+ 2125us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.185.67:43165: BlockingRecv error: recv got EOF from 127.15.185.67:43165 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260927 03:58:56.401343 25246 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 03:58:57 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.65@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:58:57 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.65@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:58:57.167428 25250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:57.162206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34493 (local address 127.15.185.65:54289)
0927 03:58:57.162343 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:57.162357 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:57.162414 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:57.162605 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:57.162605 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:57.162800 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:57.162906 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:57.162909 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:57.163781 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:57.163783 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:57.164341 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:57.164343 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:57.164733 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:57.164734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:57.164799 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:57.165312 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:57.165317 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:57.167360 (+ 2043us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.185.126:34493: Server oryx/127.15.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":91,"threads_started":1}
W20260927 03:58:57.167492 25251 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:58:57.162298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:54289 (local address 127.15.185.126:34493)
0927 03:58:57.162469 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:57.162471 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:57.162481 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:58:57.162507 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:57.162507 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:57.162517 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:57.162591 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:57.162999 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:57.163697 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:57.164429 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:57.164644 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:57.164731 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:57.167438 (+ 2707us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.185.65:54289: BlockingRecv error: recv got EOF from 127.15.185.65:54289 (error 108)
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":94,"threads_started":1}
Sep 27 03:58:57 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.66@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:58:57 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.66@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:58:57.289402 25252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:57.284478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34493 (local address 127.15.185.66:57373)
0927 03:58:57.284639 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:57.284653 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:57.284722 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:57.284849 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:57.284850 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:57.285045 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:57.285092 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:57.285093 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:57.285772 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:57.285773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:57.286330 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:57.286331 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:57.286734 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:57.286735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:57.286790 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:57.287338 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:57.287350 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:57.289276 (+ 1926us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.185.126:34493: Server oryx/127.15.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":89,"threads_started":1}
W20260927 03:58:57.289455 25251 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:58:57.284565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:57373 (local address 127.15.185.126:34493)
0927 03:58:57.284643 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:57.284646 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:57.284719 (+ 73us) server_negotiation.cc:407] Connection header received
0927 03:58:57.284777 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:57.284778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:57.284789 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:57.284836 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:57.285173 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:57.285685 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:57.286449 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:57.286660 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:57.286737 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:57.289386 (+ 2649us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.185.66:57373: BlockingRecv error: recv got EOF from 127.15.185.66:57373 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 27 03:58:57 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.67@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:58:57 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.67@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:58:57.406896 25253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:57.401796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34493 (local address 127.15.185.67:37181)
0927 03:58:57.401945 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:57.401964 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:57.402045 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:57.402195 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:57.402196 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:57.402369 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:57.402437 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:57.402438 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:57.403210 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:57.403211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:57.403755 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:57.403756 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:57.404141 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:57.404141 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:57.404199 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:57.404816 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:57.404821 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:57.406792 (+ 1971us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.185.126:34493: Server oryx/127.15.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":102,"threads_started":1}
W20260927 03:58:57.407035 25251 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:58:57.401939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:37181 (local address 127.15.185.126:34493)
0927 03:58:57.402033 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:57.402036 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:57.402046 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:58:57.402104 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:57.402105 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:57.402118 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:57.402172 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:57.402523 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:57.403121 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:57.403855 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:57.404067 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:57.404144 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:57.406917 (+ 2773us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.185.67:37181: BlockingRecv error: recv got EOF from 127.15.185.67:37181 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 27 03:58:58 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.65@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:58:58 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.65@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:58:58.172720 25254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:58.167964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34493 (local address 127.15.185.65:51387)
0927 03:58:58.168151 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:58.168162 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:58.168233 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:58.168382 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:58.168383 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:58.168542 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:58.168589 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:58.168591 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:58.169280 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:58.169281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:58.169819 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:58.169820 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:58.170207 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:58.170208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:58.170263 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:58.170825 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:58.170832 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:58.172625 (+ 1793us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.185.126:34493: Server oryx/127.15.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":105,"threads_started":1}
W20260927 03:58:58.172781 25255 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:58:58.168056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:51387 (local address 127.15.185.126:34493)
0927 03:58:58.168184 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:58.168186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:58.168217 (+ 31us) server_negotiation.cc:407] Connection header received
0927 03:58:58.168276 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:58.168277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:58.168287 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:58.168344 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:58.168654 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:58.169188 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:58.169921 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:58.170132 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:58.170216 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:58.172708 (+ 2492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.185.65:51387: BlockingRecv error: recv got EOF from 127.15.185.65:51387 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":75,"threads_started":1}
Sep 27 03:58:58 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.66@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:58:58 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.66@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:58:58.294907 25256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:58.289911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34493 (local address 127.15.185.66:44997)
0927 03:58:58.290101 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:58.290114 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:58.290208 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:58.290374 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:58.290375 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:58.290628 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:58.290685 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:58.290688 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:58.291433 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:58.291434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:58.291972 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:58.291974 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:58.292293 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:58.292294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:58.292333 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:58.292855 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:58.292860 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:58.294802 (+ 1942us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.185.126:34493: Server oryx/127.15.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":108,"threads_started":1}
W20260927 03:58:58.294940 25255 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:58:58.290038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:44997 (local address 127.15.185.126:34493)
0927 03:58:58.290105 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:58.290108 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:58.290189 (+ 81us) server_negotiation.cc:407] Connection header received
0927 03:58:58.290265 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:58.290266 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:58.290280 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:58.290333 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:58.290776 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:58.291360 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:58.292045 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:58.292227 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:58.292282 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:58.294880 (+ 2598us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.185.66:44997: BlockingRecv error: recv got EOF from 127.15.185.66:44997 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 27 03:58:58 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.67@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:58:58 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.67@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:58:58.412528 25257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:58.407463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34493 (local address 127.15.185.67:59093)
0927 03:58:58.407631 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:58.407649 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:58.407698 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:58.407885 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:58.407886 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:58.408053 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:58.408102 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:58.408104 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:58.408826 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:58.408828 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:58.409489 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:58.409491 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:58.409856 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:58.409857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:58.409918 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:58.410426 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:58.410430 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:58.412427 (+ 1997us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.185.126:34493: Server oryx/127.15.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":111,"threads_started":1}
W20260927 03:58:58.412580 25255 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:58:58.407614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:59093 (local address 127.15.185.126:34493)
0927 03:58:58.407673 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:58.407676 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:58.407693 (+ 17us) server_negotiation.cc:407] Connection header received
0927 03:58:58.407764 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:58.407765 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:58.407777 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:58.407820 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:58.408194 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:58.408708 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:58.409585 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:58.409773 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:58.409841 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:58.412515 (+ 2674us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.185.67:59093: BlockingRecv error: recv got EOF from 127.15.185.67:59093 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 27 03:58:59 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.65@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:58:59 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.65@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:58:59.177802 25258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:59.173202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34493 (local address 127.15.185.65:56143)
0927 03:58:59.173373 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:59.173389 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:59.173446 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:59.173584 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:59.173585 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:59.173741 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:59.173790 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:59.173791 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:59.174520 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:59.174521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:59.175038 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:59.175040 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:59.175452 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:59.175453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:59.175496 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:59.175994 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:59.175998 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:59.177733 (+ 1735us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.185.126:34493: Server oryx/127.15.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":102,"threads_started":1}
W20260927 03:58:59.178018 25259 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:58:59.173314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:56143 (local address 127.15.185.126:34493)
0927 03:58:59.173431 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:59.173433 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:59.173444 (+ 11us) server_negotiation.cc:407] Connection header received
0927 03:58:59.173494 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:59.173494 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:59.173505 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:59.173568 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:59.173873 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:59.174438 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:59.175190 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:59.175388 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:59.175443 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:59.177939 (+ 2496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.185.65:56143: BlockingRecv error: recv got EOF from 127.15.185.65:56143 (error 108)
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":63,"threads_started":1}
Sep 27 03:58:59 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.66@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:58:59 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.66@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:58:59.300417 25260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:59.295502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34493 (local address 127.15.185.66:57739)
0927 03:58:59.295688 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:59.295699 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:59.295786 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:59.295951 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:59.295952 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:59.296131 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:59.296176 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:59.296177 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:59.296862 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:59.296864 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:59.297421 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:59.297423 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:59.297805 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:59.297806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:59.297862 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:59.298487 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:59.298492 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:59.300359 (+ 1867us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.185.126:34493: Server oryx/127.15.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":113,"threads_started":1}
W20260927 03:58:59.300531 25259 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:58:59.295598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:57739 (local address 127.15.185.126:34493)
0927 03:58:59.295668 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:59.295671 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:59.295767 (+ 96us) server_negotiation.cc:407] Connection header received
0927 03:58:59.295856 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:59.295858 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:59.295874 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:59.295945 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:59.296261 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:59.296761 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:59.297523 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:59.297720 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:59.297805 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:59.300467 (+ 2662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.185.66:57739: BlockingRecv error: recv got EOF from 127.15.185.66:57739 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 27 03:58:59 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.67@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:58:59 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.67@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:58:59.418416 25261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:58:59.413191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34493 (local address 127.15.185.67:45079)
0927 03:58:59.413365 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 03:58:59.413387 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 03:58:59.413487 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:58:59.413663 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:58:59.413664 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:58:59.413871 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:58:59.413920 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:59.413922 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:59.414619 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:59.414620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:59.415249 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:58:59.415251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:58:59.415622 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:58:59.415623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:58:59.415675 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:59.416216 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:58:59.416221 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:58:59.418311 (+ 2090us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.185.126:34493: Server oryx/127.15.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":102,"threads_started":1}
W20260927 03:58:59.418570 25259 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:58:59.413344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:45079 (local address 127.15.185.126:34493)
0927 03:58:59.413442 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 03:58:59.413445 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:58:59.413461 (+ 16us) server_negotiation.cc:407] Connection header received
0927 03:58:59.413554 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:58:59.413555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:58:59.413570 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:58:59.413654 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:58:59.414030 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:59.414530 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:59.415331 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:58:59.415523 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:58:59.415625 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:58:59.418412 (+ 2787us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.185.67:45079: BlockingRecv error: recv got EOF from 127.15.185.67:45079 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 27 03:59:00 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.65@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:59:00 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.65@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:59:00.183311 25262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:00.178372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34493 (local address 127.15.185.65:35017)
0927 03:59:00.178520 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 03:59:00.178531 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:59:00.178589 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:59:00.178739 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:59:00.178740 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:59:00.178903 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:59:00.178954 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:00.178956 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:00.179644 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:00.179645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:00.180230 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:00.180232 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:00.180597 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:00.180598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:00.180637 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:00.181115 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:59:00.181120 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:59:00.183247 (+ 2127us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.185.126:34493: Server oryx/127.15.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":87,"threads_started":1}
W20260927 03:59:00.183503 25263 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:59:00.178440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:35017 (local address 127.15.185.126:34493)
0927 03:59:00.178567 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 03:59:00.178569 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:59:00.178579 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:59:00.178642 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:59:00.178643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:59:00.178653 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:59:00.178719 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:59:00.179025 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:00.179560 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:00.180330 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:00.180526 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:00.180588 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:00.183377 (+ 2789us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.185.65:35017: BlockingRecv error: recv got EOF from 127.15.185.65:35017 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":74,"threads_started":1}
Sep 27 03:59:00 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.66@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:59:00 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.66@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:59:00.310457 25264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:00.300985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34493 (local address 127.15.185.66:52873)
0927 03:59:00.301201 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 03:59:00.301215 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 03:59:00.301278 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:59:00.301475 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:59:00.301476 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:59:00.301682 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:59:00.301740 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:00.301742 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:00.302425 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:00.302426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:00.302951 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:00.302952 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:00.303368 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:00.303369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:00.303432 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:00.308467 (+ 5035us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:59:00.308473 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:59:00.310396 (+ 1923us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.185.126:34493: Server oryx/127.15.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":92,"threads_started":1}
W20260927 03:59:00.310606 25263 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:59:00.301095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:52873 (local address 127.15.185.126:34493)
0927 03:59:00.301252 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 03:59:00.301254 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:59:00.301282 (+ 28us) server_negotiation.cc:407] Connection header received
0927 03:59:00.301354 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:59:00.301355 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:59:00.301367 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:59:00.301431 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:59:00.301816 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:00.302334 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:00.303027 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:00.303304 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:00.303362 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:00.310514 (+ 7152us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.185.66:52873: BlockingRecv error: recv got EOF from 127.15.185.66:52873 (error 108)
Metrics: {"server-negotiator.queue_time_us":130}
Sep 27 03:59:00 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.67@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:59:00 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.67@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:59:00.424305 25265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:00.418964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34493 (local address 127.15.185.67:56085)
0927 03:59:00.419108 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 03:59:00.419124 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 03:59:00.419216 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:59:00.419372 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:59:00.419373 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:59:00.419589 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:59:00.419637 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:00.419640 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:00.420344 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:00.420346 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:00.421125 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:00.421129 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:00.421479 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:00.421480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:00.421540 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:00.422022 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:59:00.422027 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:59:00.424199 (+ 2172us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.185.126:34493: Server oryx/127.15.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":66,"threads_started":1}
W20260927 03:59:00.424363 25263 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:59:00.419109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:56085 (local address 127.15.185.126:34493)
0927 03:59:00.419186 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0927 03:59:00.419188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:59:00.419198 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:59:00.419305 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:59:00.419306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:59:00.419319 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:59:00.419368 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:59:00.419722 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:00.420237 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:00.421190 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:00.421384 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:00.421476 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:00.424288 (+ 2812us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.185.67:56085: BlockingRecv error: recv got EOF from 127.15.185.67:56085 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 27 03:59:01 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.65@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:59:01 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.65@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:59:01.188709 25266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:01.183895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34493 (local address 127.15.185.65:51135)
0927 03:59:01.184157 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0927 03:59:01.184169 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:59:01.184226 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:59:01.184360 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:59:01.184360 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:59:01.184523 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:59:01.184572 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:01.184574 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:01.185270 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:01.185270 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:01.185816 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:01.185818 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:01.186183 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:01.186183 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:01.186225 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:01.186659 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:59:01.186663 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:59:01.188650 (+ 1987us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.185.126:34493: Server oryx/127.15.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":189,"threads_started":1}
W20260927 03:59:01.188798 25267 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:59:01.184092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:51135 (local address 127.15.185.126:34493)
0927 03:59:01.184236 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 03:59:01.184238 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:59:01.184246 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:59:01.184271 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:59:01.184272 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:59:01.184282 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:59:01.184344 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:59:01.184660 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:01.185197 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:01.185927 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:01.186115 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:01.186171 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:01.188745 (+ 2574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.185.65:51135: BlockingRecv error: recv got EOF from 127.15.185.65:51135 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":89,"threads_started":1}
Sep 27 03:59:01 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.66@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:59:01 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.66@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:59:01.315935 25268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:01.310976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34493 (local address 127.15.185.66:43601)
0927 03:59:01.311138 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 03:59:01.311150 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:59:01.311209 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:59:01.311512 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:59:01.311513 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:59:01.311691 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:59:01.311735 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:01.311737 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:01.312443 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:01.312444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:01.312977 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:01.312978 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:01.313374 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:01.313375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:01.313442 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:01.314010 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:59:01.314014 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:59:01.315843 (+ 1829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.185.126:34493: Server oryx/127.15.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":128,"threads_started":1}
W20260927 03:59:01.316016 25267 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:59:01.311304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:43601 (local address 127.15.185.126:34493)
0927 03:59:01.311383 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0927 03:59:01.311386 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:59:01.311394 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:59:01.311426 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:59:01.311427 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:59:01.311437 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:59:01.311504 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:59:01.311821 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:01.312356 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:01.313075 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:01.313279 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:01.313372 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:01.315951 (+ 2579us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.185.66:43601: BlockingRecv error: recv got EOF from 127.15.185.66:43601 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 27 03:59:01 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.67@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:59:01 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.67@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:59:01.430079 25269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:01.424877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34493 (local address 127.15.185.67:40337)
0927 03:59:01.425118 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 03:59:01.425133 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 03:59:01.425235 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:59:01.425437 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:59:01.425438 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:59:01.425615 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:59:01.425662 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:01.425664 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:01.426348 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:01.426349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:01.426872 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:01.426874 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:01.427333 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:01.427334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:01.427400 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:01.428031 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:59:01.428037 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:59:01.430022 (+ 1985us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.185.126:34493: Server oryx/127.15.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":135,"threads_started":1}
W20260927 03:59:01.430287 25267 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:59:01.425007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:40337 (local address 127.15.185.126:34493)
0927 03:59:01.425102 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0927 03:59:01.425105 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:59:01.425214 (+ 109us) server_negotiation.cc:407] Connection header received
0927 03:59:01.425323 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:59:01.425324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:59:01.425339 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:59:01.425426 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:59:01.425745 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:01.426254 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:01.426980 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:01.427244 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:01.427333 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:01.430184 (+ 2851us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.185.67:40337: BlockingRecv error: recv got EOF from 127.15.185.67:40337 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 27 03:59:02 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.65@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:59:02 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.65@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:59:02.194629 25270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:02.189411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34493 (local address 127.15.185.65:40849)
0927 03:59:02.189564 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 03:59:02.189581 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 03:59:02.189670 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:59:02.189889 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:59:02.189890 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:59:02.190115 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:59:02.190164 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:02.190166 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:02.190902 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:02.190903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:02.191726 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:02.191728 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:02.192044 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:02.192045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:02.192084 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:02.192591 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:59:02.192596 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:59:02.194561 (+ 1965us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.185.126:34493: Server oryx/127.15.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":97,"threads_started":1}
W20260927 03:59:02.194700 25271 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:59:02.189507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:40849 (local address 127.15.185.126:34493)
0927 03:59:02.189631 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 03:59:02.189632 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:59:02.189644 (+ 12us) server_negotiation.cc:407] Connection header received
0927 03:59:02.189746 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:59:02.189746 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:59:02.189757 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:59:02.189827 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:59:02.190232 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:02.190836 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:02.191789 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:02.191981 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:02.192036 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:02.194647 (+ 2611us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.185.65:40849: BlockingRecv error: recv got EOF from 127.15.185.65:40849 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":78,"threads_started":1}
Sep 27 03:59:02 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.66@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:59:02 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.66@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:59:02.321238 25272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:02.316460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34493 (local address 127.15.185.66:42397)
0927 03:59:02.316706 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 03:59:02.316811 (+ 105us) client_negotiation.cc:174] Beginning negotiation
0927 03:59:02.316884 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:59:02.317054 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:59:02.317055 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:59:02.317215 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:59:02.317266 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:02.317268 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:02.317907 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:02.317908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:02.318455 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:02.318457 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:02.318815 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:02.318818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:02.318870 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:02.319388 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:59:02.319393 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:59:02.321178 (+ 1785us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.185.126:34493: Server oryx/127.15.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":193,"threads_started":1}
W20260927 03:59:02.321337 25271 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:59:02.316635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:42397 (local address 127.15.185.126:34493)
0927 03:59:02.316681 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 03:59:02.316683 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:59:02.316879 (+ 196us) server_negotiation.cc:407] Connection header received
0927 03:59:02.316940 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:59:02.316941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:59:02.316952 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:59:02.317005 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:59:02.317355 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:02.317836 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:02.318524 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:02.318717 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:02.318774 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:02.321270 (+ 2496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.185.66:42397: BlockingRecv error: recv got EOF from 127.15.185.66:42397 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Sep 27 03:59:02 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.67@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:59:02 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.67@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:59:02.435518 25273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:02.430662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34493 (local address 127.15.185.67:36017)
0927 03:59:02.430855 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 03:59:02.430869 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 03:59:02.430945 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:59:02.431114 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:59:02.431115 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:59:02.431279 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:59:02.431331 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:02.431333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:02.432025 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:02.432027 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:02.432566 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:02.432568 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:02.432918 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:02.432919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:02.433000 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:02.433604 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:59:02.433609 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:59:02.435460 (+ 1851us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.185.126:34493: Server oryx/127.15.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":89,"threads_started":1}
W20260927 03:59:02.435658 25271 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:59:02.430740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:36017 (local address 127.15.185.126:34493)
0927 03:59:02.430856 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 03:59:02.430859 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:59:02.430947 (+ 88us) server_negotiation.cc:407] Connection header received
0927 03:59:02.431014 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:59:02.431015 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:59:02.431029 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:59:02.431094 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:59:02.431397 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:02.431888 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:02.432650 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:02.432833 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:02.432889 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:02.435571 (+ 2682us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.185.67:36017: BlockingRecv error: recv got EOF from 127.15.185.67:36017 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 27 03:59:03 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.65@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:59:03 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.65@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:59:03.200245 25274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:03.195313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34493 (local address 127.15.185.65:60507)
0927 03:59:03.195494 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 03:59:03.195504 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:59:03.195565 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:59:03.195706 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:59:03.195706 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:59:03.195867 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:59:03.195923 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:03.195925 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:03.196669 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:03.196670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:03.197192 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:03.197193 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:03.197552 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:03.197552 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:03.197591 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:03.198053 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:59:03.198057 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:59:03.200177 (+ 2120us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.185.126:34493: Server oryx/127.15.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":98,"threads_started":1}
W20260927 03:59:03.200500 25275 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:59:03.195363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:60507 (local address 127.15.185.126:34493)
0927 03:59:03.195517 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 03:59:03.195519 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:59:03.195549 (+ 30us) server_negotiation.cc:407] Connection header received
0927 03:59:03.195617 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:59:03.195618 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:59:03.195628 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:59:03.195687 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:59:03.196020 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:03.196594 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:03.197265 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:03.197483 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:03.197543 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:03.200291 (+ 2748us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.185.65:60507: BlockingRecv error: recv got EOF from 127.15.185.65:60507 (error 108)
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":79,"threads_started":1}
Sep 27 03:59:03 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.66@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:59:03 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.66@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:59:03.326462 25276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:03.321718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34493 (local address 127.15.185.66:48331)
0927 03:59:03.321896 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 03:59:03.321908 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:59:03.321981 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:59:03.322158 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:59:03.322159 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:59:03.322335 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:59:03.322379 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:03.322380 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:03.323089 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:03.323090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:03.323635 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:03.323636 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:03.324032 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:03.324033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:03.324092 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:03.324557 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:59:03.324562 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:59:03.326356 (+ 1794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.185.126:34493: Server oryx/127.15.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":87,"threads_started":1}
W20260927 03:59:03.326505 25275 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:59:03.321802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:48331 (local address 127.15.185.126:34493)
0927 03:59:03.321871 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 03:59:03.321874 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:59:03.321960 (+ 86us) server_negotiation.cc:407] Connection header received
0927 03:59:03.322045 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:59:03.322046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:59:03.322059 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:59:03.322145 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:59:03.322469 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:03.322971 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:03.323737 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:03.323943 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:03.324031 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:03.326439 (+ 2408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.185.66:48331: BlockingRecv error: recv got EOF from 127.15.185.66:48331 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 27 03:59:03 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.67@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:59:03 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.67@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:59:03.440757 25277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:03.435964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34493 (local address 127.15.185.67:53037)
0927 03:59:03.436154 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 03:59:03.436166 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:59:03.436259 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:59:03.436407 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:59:03.436408 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:59:03.436609 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:59:03.436655 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:03.436657 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:03.437354 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:03.437355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:03.438036 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:03.438039 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:03.438394 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:03.438395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:03.438451 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:03.438889 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:59:03.438893 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:59:03.440699 (+ 1806us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.185.126:34493: Server oryx/127.15.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":100,"threads_started":1}
W20260927 03:59:03.440939 25275 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:59:03.436033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:53037 (local address 127.15.185.126:34493)
0927 03:59:03.436129 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0927 03:59:03.436132 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:59:03.436244 (+ 112us) server_negotiation.cc:407] Connection header received
0927 03:59:03.436319 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:59:03.436320 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:59:03.436334 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:59:03.436390 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:59:03.436738 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:03.437259 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:03.438120 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:03.438309 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:03.438392 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:03.440805 (+ 2413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.185.67:53037: BlockingRecv error: recv got EOF from 127.15.185.67:53037 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 27 03:59:04 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.65@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:59:04 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.65@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:59:04.206162 25278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:04.200961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34493 (local address 127.15.185.65:60089)
0927 03:59:04.201110 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 03:59:04.201121 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:59:04.201174 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:59:04.201395 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:59:04.201396 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:59:04.201563 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:59:04.201614 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:04.201615 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:04.202649 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:04.202650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:04.203262 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:04.203264 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:04.203630 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:04.203631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:04.203669 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:04.204197 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:59:04.204202 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:59:04.206096 (+ 1894us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.185.126:34493: Server oryx/127.15.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":90,"threads_started":1}
W20260927 03:59:04.206271 25279 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:59:04.201123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:60089 (local address 127.15.185.126:34493)
0927 03:59:04.201256 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 03:59:04.201258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:59:04.201267 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:59:04.201291 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:59:04.201292 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:59:04.201303 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:59:04.201376 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:59:04.201695 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:04.202507 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:04.203362 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:04.203559 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:04.203620 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:04.206195 (+ 2575us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.185.65:60089: BlockingRecv error: recv got EOF from 127.15.185.65:60089 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":74,"threads_started":1}
Sep 27 03:59:04 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.66@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:59:04 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.66@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:59:04.332198 25280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:04.327148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34493 (local address 127.15.185.66:52545)
0927 03:59:04.327302 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 03:59:04.327314 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:59:04.327407 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:59:04.327570 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:59:04.327571 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:59:04.327753 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:59:04.327798 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:04.327800 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:04.328621 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:04.328622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:04.329156 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:04.329158 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:04.329536 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:04.329537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:04.329591 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:04.330079 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:59:04.330084 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:59:04.332091 (+ 2007us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.185.126:34493: Server oryx/127.15.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":64,"threads_started":1}
W20260927 03:59:04.332234 25279 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:59:04.327165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:52545 (local address 127.15.185.126:34493)
0927 03:59:04.327287 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 03:59:04.327289 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:59:04.327387 (+ 98us) server_negotiation.cc:407] Connection header received
0927 03:59:04.327478 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:59:04.327479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:59:04.327492 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:59:04.327565 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:59:04.327885 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:04.328537 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:04.329258 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:04.329461 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:04.329539 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:04.332168 (+ 2629us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.185.66:52545: BlockingRecv error: recv got EOF from 127.15.185.66:52545 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 27 03:59:04 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.67@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:59:04 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.67@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:59:04.445961 25281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:04.441221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34493 (local address 127.15.185.67:40073)
0927 03:59:04.441356 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 03:59:04.441378 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 03:59:04.441433 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:59:04.441563 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:59:04.441564 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:59:04.441743 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:59:04.441792 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:04.441794 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:04.442428 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:04.442429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:04.443236 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:04.443240 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:04.443614 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:04.443615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:04.443670 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:04.444102 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:59:04.444107 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:59:04.445890 (+ 1783us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.185.126:34493: Server oryx/127.15.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":84,"threads_started":1}
W20260927 03:59:04.446110 25279 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:59:04.441298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:40073 (local address 127.15.185.126:34493)
0927 03:59:04.441364 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 03:59:04.441365 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 03:59:04.441418 (+ 53us) server_negotiation.cc:407] Connection header received
0927 03:59:04.441488 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:59:04.441489 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:59:04.441499 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:59:04.441542 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:59:04.441853 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:04.442344 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:04.443337 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:04.443530 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:04.443613 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:04.446028 (+ 2415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.185.67:40073: BlockingRecv error: recv got EOF from 127.15.185.67:40073 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 27 03:59:05 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.65@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:59:05 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.65@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:59:05.211824 25282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:05.206733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34493 (local address 127.15.185.65:52695)
0927 03:59:05.206899 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 03:59:05.206913 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 03:59:05.206980 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:59:05.207193 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:59:05.207194 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:59:05.207362 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:59:05.207412 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:05.207414 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:05.208122 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:05.208123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:05.208876 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:05.208879 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:05.209206 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:05.209207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:05.209251 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:05.209722 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:59:05.209726 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:59:05.211764 (+ 2038us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.185.126:34493: Server oryx/127.15.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":168,"threads_started":1}
W20260927 03:59:05.211943 25283 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:59:05.206829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:52695 (local address 127.15.185.126:34493)
0927 03:59:05.206948 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 03:59:05.206950 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:59:05.206967 (+ 17us) server_negotiation.cc:407] Connection header received
0927 03:59:05.207030 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:59:05.207031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:59:05.207088 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:59:05.207170 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:59:05.207485 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:05.208025 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:05.208945 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:05.209135 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:05.209189 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:05.211859 (+ 2670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.185.65:52695: BlockingRecv error: recv got EOF from 127.15.185.65:52695 (error 108)
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":71,"threads_started":1}
Sep 27 03:59:05 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.66@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:59:05 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.66@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:59:05.342381 25284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:05.332650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34493 (local address 127.15.185.66:34367)
0927 03:59:05.332832 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 03:59:05.332843 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:59:05.332915 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:59:05.333066 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:59:05.333067 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:59:05.333253 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:59:05.333296 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:05.333298 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:05.334023 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:05.334025 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:05.334565 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:05.334566 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:05.334927 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:05.334928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:05.334981 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:05.340144 (+ 5163us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:59:05.340148 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:59:05.342317 (+ 2169us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.185.126:34493: Server oryx/127.15.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":87,"threads_started":1}
W20260927 03:59:05.342538 25283 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:59:05.332741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:34367 (local address 127.15.185.126:34493)
0927 03:59:05.332808 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 03:59:05.332810 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:59:05.332914 (+ 104us) server_negotiation.cc:407] Connection header received
0927 03:59:05.332976 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:59:05.332977 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:59:05.332990 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:59:05.333066 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:59:05.333376 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:05.333889 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:05.334651 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:05.334843 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:05.334933 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:05.342443 (+ 7510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.185.66:34367: BlockingRecv error: recv got EOF from 127.15.185.66:34367 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 27 03:59:05 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.67@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:59:05 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.67@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:59:05.451370 25285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:05.446464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34493 (local address 127.15.185.67:55637)
0927 03:59:05.446624 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 03:59:05.446642 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 03:59:05.446704 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:59:05.446883 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:59:05.446884 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:59:05.447072 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:59:05.447123 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:05.447125 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:05.447791 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:05.447793 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:05.448347 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:05.448349 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:05.448705 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:05.448706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:05.448748 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:05.449183 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:59:05.449187 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:59:05.451277 (+ 2090us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.185.126:34493: Server oryx/127.15.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":102,"threads_started":1}
W20260927 03:59:05.451406 25283 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:59:05.446548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:55637 (local address 127.15.185.126:34493)
0927 03:59:05.446617 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 03:59:05.446619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:59:05.446702 (+ 83us) server_negotiation.cc:407] Connection header received
0927 03:59:05.446748 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:59:05.446749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:59:05.446759 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:59:05.446815 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:59:05.447198 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:05.447698 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:05.448453 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:05.448642 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:05.448698 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:05.451345 (+ 2647us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.185.67:55637: BlockingRecv error: recv got EOF from 127.15.185.67:55637 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 27 03:59:06 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.65@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:59:06 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.65@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:59:06.219200 25287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:06.212415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34493 (local address 127.15.185.65:59087)
0927 03:59:06.212637 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0927 03:59:06.212650 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:59:06.212733 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:59:06.212936 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:59:06.212938 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:59:06.213157 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:59:06.213226 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:06.213229 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:06.214397 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:06.214398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:06.215392 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:06.215396 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:06.216001 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:06.216003 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:06.216066 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:06.216630 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:59:06.216636 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:59:06.219100 (+ 2464us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.185.126:34493: Server oryx/127.15.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":157,"threads_started":1}
W20260927 03:59:06.219257 25286 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:59:06.212486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:59087 (local address 127.15.185.126:34493)
0927 03:59:06.212640 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 03:59:06.212656 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 03:59:06.212712 (+ 56us) server_negotiation.cc:407] Connection header received
0927 03:59:06.212798 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:59:06.212799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:59:06.212812 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:59:06.212918 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:59:06.213321 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:06.214242 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:06.215488 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:06.215823 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:06.215903 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:06.219183 (+ 3280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.185.65:59087: BlockingRecv error: recv got EOF from 127.15.185.65:59087 (error 108)
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":59,"threads_started":1}
Sep 27 03:59:06 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.66@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:59:06 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.66@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:59:06.347831 25288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:06.342947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34493 (local address 127.15.185.66:48179)
0927 03:59:06.343196 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 03:59:06.343207 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:59:06.343270 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:59:06.343402 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:59:06.343403 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:59:06.343568 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:59:06.343615 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:06.343617 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:06.344337 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:06.344338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:06.344867 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:06.344868 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:06.345216 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:06.345217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:06.345256 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:06.345714 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:59:06.345718 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:59:06.347761 (+ 2043us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.185.126:34493: Server oryx/127.15.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":182,"threads_started":1}
W20260927 03:59:06.348033 25286 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:59:06.343084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:48179 (local address 127.15.185.126:34493)
0927 03:59:06.343155 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 03:59:06.343157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:59:06.343276 (+ 119us) server_negotiation.cc:407] Connection header received
0927 03:59:06.343330 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:59:06.343331 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:59:06.343341 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:59:06.343385 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:59:06.343724 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:06.344259 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:06.344962 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:06.345149 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:06.345203 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:06.347965 (+ 2762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.185.66:48179: BlockingRecv error: recv got EOF from 127.15.185.66:48179 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 27 03:59:06 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.67@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:59:06 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.67@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:59:06.456581 25289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:06.451894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34493 (local address 127.15.185.67:51901)
0927 03:59:06.452052 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 03:59:06.452065 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 03:59:06.452130 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:59:06.452280 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:59:06.452280 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:59:06.452468 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:59:06.452516 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:06.452517 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:06.453182 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:06.453183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:06.453713 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:06.453715 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:06.454075 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:06.454076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:06.454134 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:06.454605 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:59:06.454610 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:59:06.456524 (+ 1914us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.185.126:34493: Server oryx/127.15.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":73,"threads_started":1}
W20260927 03:59:06.456681 25286 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:59:06.452050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:51901 (local address 127.15.185.126:34493)
0927 03:59:06.452103 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 03:59:06.452105 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:59:06.452115 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:59:06.452189 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:59:06.452190 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:59:06.452205 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:59:06.452274 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:59:06.452602 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:06.453095 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:06.453812 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:06.454001 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:06.454077 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:06.456613 (+ 2536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.185.67:51901: BlockingRecv error: recv got EOF from 127.15.185.67:51901 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 27 03:59:07 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.65@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:59:07 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.65@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:59:07.224521 25290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:07.219822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34493 (local address 127.15.185.65:37929)
0927 03:59:07.219956 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 03:59:07.219973 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 03:59:07.220026 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:59:07.220211 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:59:07.220212 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:59:07.220393 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:59:07.220440 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:07.220441 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:07.221110 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:07.221111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:07.221691 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:07.221693 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:07.222043 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:07.222044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:07.222095 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:07.222590 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:59:07.222594 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:59:07.224451 (+ 1857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.185.126:34493: Server oryx/127.15.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":79,"threads_started":1}
W20260927 03:59:07.224694 25291 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:59:07.219905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:37929 (local address 127.15.185.126:34493)
0927 03:59:07.220050 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 03:59:07.220052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:59:07.220060 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:59:07.220080 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:59:07.220081 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:59:07.220091 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:59:07.220170 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:59:07.220513 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:07.221018 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:07.221765 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:07.221948 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:07.222012 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:07.224599 (+ 2587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.185.65:37929: BlockingRecv error: recv got EOF from 127.15.185.65:37929 (error 108)
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":81,"threads_started":1}
Sep 27 03:59:07 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.66@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:59:07 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.66@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:59:07.353040 25292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:07.348408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34493 (local address 127.15.185.66:59749)
0927 03:59:07.348529 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 03:59:07.348541 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:59:07.348614 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:59:07.348823 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:59:07.348824 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:59:07.349052 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:59:07.349098 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:07.349100 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:07.349801 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:07.349801 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:07.350353 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:07.350355 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:07.350717 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:07.350718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:07.350763 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:07.351243 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:59:07.351247 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:59:07.352970 (+ 1723us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.185.126:34493: Server oryx/127.15.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":72,"threads_started":1}
W20260927 03:59:07.353117 25291 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:59:07.348457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:59749 (local address 127.15.185.126:34493)
0927 03:59:07.348526 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 03:59:07.348528 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:59:07.348620 (+ 92us) server_negotiation.cc:407] Connection header received
0927 03:59:07.348691 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:59:07.348692 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:59:07.348704 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:59:07.348784 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:59:07.349171 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:07.349728 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:07.350433 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:07.350629 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:07.350684 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:07.353069 (+ 2385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.185.66:59749: BlockingRecv error: recv got EOF from 127.15.185.66:59749 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 27 03:59:07 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.67@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:59:07 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.67@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:59:07.461861 25293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:07.457088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34493 (local address 127.15.185.67:46625)
0927 03:59:07.457265 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 03:59:07.457283 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 03:59:07.457341 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:59:07.457475 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:59:07.457476 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:59:07.457646 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:59:07.457692 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:07.457693 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:07.458477 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:07.458478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:07.459077 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:07.459079 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:07.459386 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:07.459387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:07.459423 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:07.459911 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:59:07.459916 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:59:07.461741 (+ 1825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.185.126:34493: Server oryx/127.15.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":133,"threads_started":1}
W20260927 03:59:07.461877 25291 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:59:07.457185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:46625 (local address 127.15.185.126:34493)
0927 03:59:07.457254 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 03:59:07.457260 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 03:59:07.457345 (+ 85us) server_negotiation.cc:407] Connection header received
0927 03:59:07.457385 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:59:07.457385 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:59:07.457396 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:59:07.457458 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:59:07.457760 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:07.458410 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:07.459150 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:07.459324 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:07.459375 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:07.461829 (+ 2454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.185.67:46625: BlockingRecv error: recv got EOF from 127.15.185.67:46625 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 27 03:59:08 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.65@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:59:08 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.65@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:59:08.230595 25294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:08.225095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34493 (local address 127.15.185.65:56555)
0927 03:59:08.225326 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0927 03:59:08.225340 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 03:59:08.225415 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:59:08.225617 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:59:08.225618 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:59:08.225903 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:59:08.225971 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:08.225974 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:08.226672 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:08.226673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:08.227372 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:08.227375 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:08.227744 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:08.227746 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:08.227803 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:08.228475 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:59:08.228489 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:59:08.230535 (+ 2046us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.185.126:34493: Server oryx/127.15.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":111,"threads_started":1}
W20260927 03:59:08.230692 25295 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:59:08.225205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:56555 (local address 127.15.185.126:34493)
0927 03:59:08.225398 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 03:59:08.225401 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:59:08.225414 (+ 13us) server_negotiation.cc:407] Connection header received
0927 03:59:08.225498 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:59:08.225499 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:59:08.225512 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:59:08.225607 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:59:08.226064 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:08.226589 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:08.227455 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:08.227665 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:08.227722 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:08.230610 (+ 2888us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.185.65:56555: BlockingRecv error: recv got EOF from 127.15.185.65:56555 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":55,"threads_started":1}
Sep 27 03:59:08 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.66@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:59:08 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.66@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:59:08.358613 25296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:08.353661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34493 (local address 127.15.185.66:50811)
0927 03:59:08.353830 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 03:59:08.353840 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:59:08.353899 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:59:08.354109 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:59:08.354110 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:59:08.354273 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:59:08.354316 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:08.354318 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:08.355155 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:08.355156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:08.355680 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:08.355681 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:08.356082 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:08.356083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:08.356162 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:08.356685 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:59:08.356690 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:59:08.358517 (+ 1827us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.185.126:34493: Server oryx/127.15.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":95,"threads_started":1}
W20260927 03:59:08.358649 25295 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:59:08.353750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:50811 (local address 127.15.185.126:34493)
0927 03:59:08.353810 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 03:59:08.353812 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:59:08.353898 (+ 86us) server_negotiation.cc:407] Connection header received
0927 03:59:08.354018 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:59:08.354019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:59:08.354031 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:59:08.354088 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:59:08.354383 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:08.355073 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:08.355749 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:08.355988 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:08.356065 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:08.358590 (+ 2525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.185.66:50811: BlockingRecv error: recv got EOF from 127.15.185.66:50811 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 27 03:59:08 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.67@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:59:08 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.67@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:59:08.467130 25297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:08.462407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34493 (local address 127.15.185.67:48183)
0927 03:59:08.462547 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 03:59:08.462565 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 03:59:08.462618 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:59:08.462745 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:59:08.462746 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:59:08.462910 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:59:08.462954 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:08.462962 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:08.463616 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:08.463616 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:08.464181 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:08.464183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:08.464531 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:08.464532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:08.464587 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:08.465209 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:59:08.465215 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:59:08.467002 (+ 1787us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.185.126:34493: Server oryx/127.15.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":90,"threads_started":1}
W20260927 03:59:08.467187 25295 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:59:08.462492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:48183 (local address 127.15.185.126:34493)
0927 03:59:08.462553 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 03:59:08.462555 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:59:08.462605 (+ 50us) server_negotiation.cc:407] Connection header received
0927 03:59:08.462674 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:59:08.462675 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:59:08.462685 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:59:08.462727 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:59:08.463035 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:08.463550 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:08.464261 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:08.464452 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:08.464510 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:08.467140 (+ 2630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.185.67:48183: BlockingRecv error: recv got EOF from 127.15.185.67:48183 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 27 03:59:09 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.65@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:59:09 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.65@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:59:09.235918 25298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:09.231153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34493 (local address 127.15.185.65:33185)
0927 03:59:09.231283 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 03:59:09.231293 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:59:09.231339 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:59:09.231472 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:59:09.231473 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:59:09.231677 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:59:09.231722 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:09.231724 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:09.232439 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:09.232440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:09.233000 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:09.233002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:09.233369 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:09.233370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:09.233410 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:09.233874 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:59:09.233878 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:59:09.235845 (+ 1967us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.185.126:34493: Server oryx/127.15.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":71,"threads_started":1}
W20260927 03:59:09.236088 25299 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:59:09.231190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:33185 (local address 127.15.185.126:34493)
0927 03:59:09.231315 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 03:59:09.231317 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:59:09.231341 (+ 24us) server_negotiation.cc:407] Connection header received
0927 03:59:09.231398 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:59:09.231398 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:59:09.231409 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:59:09.231461 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:59:09.231790 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:09.232343 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:09.233081 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:09.233302 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:09.233362 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:09.235973 (+ 2611us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.185.65:33185: BlockingRecv error: recv got EOF from 127.15.185.65:33185 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":63,"threads_started":1}
Sep 27 03:59:09 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.66@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:59:09 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.66@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:59:09.363957 25300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:09.359143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34493 (local address 127.15.185.66:52819)
0927 03:59:09.359284 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 03:59:09.359300 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 03:59:09.359346 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:59:09.359513 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:59:09.359513 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:59:09.359716 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:59:09.359760 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:09.359761 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:09.360735 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:09.360736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:09.361308 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:09.361310 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:09.361655 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:09.361656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:09.361694 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:09.362155 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:59:09.362160 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:59:09.363890 (+ 1730us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.185.126:34493: Server oryx/127.15.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":83,"threads_started":1}
W20260927 03:59:09.364123 25299 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:59:09.359306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:52819 (local address 127.15.185.126:34493)
0927 03:59:09.359372 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 03:59:09.359374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:59:09.359386 (+ 12us) server_negotiation.cc:407] Connection header received
0927 03:59:09.359411 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:59:09.359412 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:59:09.359421 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:59:09.359479 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:59:09.359902 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:09.360663 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:09.361404 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:09.361586 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:09.361646 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:09.364038 (+ 2392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.185.66:52819: BlockingRecv error: recv got EOF from 127.15.185.66:52819 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 27 03:59:09 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.67@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:59:09 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.67@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:59:09.472389 25301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:09.467699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34493 (local address 127.15.185.67:58041)
0927 03:59:09.467835 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 03:59:09.467846 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:59:09.467911 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:59:09.468045 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:59:09.468045 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:59:09.468228 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:59:09.468271 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:09.468273 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:09.468976 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:09.468977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:09.469509 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:09.469510 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:09.469937 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:09.469938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:09.469998 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:09.470527 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:59:09.470537 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:59:09.472277 (+ 1740us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.185.126:34493: Server oryx/127.15.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":84,"threads_started":1}
W20260927 03:59:09.472410 25299 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:59:09.467750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:58041 (local address 127.15.185.126:34493)
0927 03:59:09.467813 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 03:59:09.467815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:59:09.467915 (+ 100us) server_negotiation.cc:407] Connection header received
0927 03:59:09.467976 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:59:09.467977 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:59:09.467990 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:59:09.468041 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:59:09.468360 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:09.468890 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:09.469595 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:09.469861 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:09.469929 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:09.472355 (+ 2426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.185.67:58041: BlockingRecv error: recv got EOF from 127.15.185.67:58041 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 27 03:59:10 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.65@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:59:10 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.65@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:59:10.241087 25302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:10.236402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34493 (local address 127.15.185.65:42475)
0927 03:59:10.236537 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 03:59:10.236556 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 03:59:10.236607 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:59:10.236779 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:59:10.236780 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:59:10.236964 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:59:10.237014 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:10.237016 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:10.237699 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:10.237700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:10.238251 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:10.238253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:10.238615 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:10.238616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:10.238663 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:10.239160 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:59:10.239164 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:59:10.241026 (+ 1862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.185.126:34493: Server oryx/127.15.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":82,"threads_started":1}
W20260927 03:59:10.241230 25303 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:59:10.236491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:42475 (local address 127.15.185.126:34493)
0927 03:59:10.236617 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 03:59:10.236619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:59:10.236627 (+ 8us) server_negotiation.cc:407] Connection header received
0927 03:59:10.236649 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:59:10.236649 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:59:10.236659 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:59:10.236734 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:59:10.237095 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:10.237613 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:10.238332 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:10.238540 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:10.238595 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:10.241136 (+ 2541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.185.65:42475: BlockingRecv error: recv got EOF from 127.15.185.65:42475 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":70,"threads_started":1}
Sep 27 03:59:10 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.66@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:59:10 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.66@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:59:10.374646 25304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:10.364494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34493 (local address 127.15.185.66:42957)
0927 03:59:10.364661 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 03:59:10.364673 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:59:10.364720 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:59:10.364858 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:59:10.364859 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:59:10.365030 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:59:10.365077 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:10.365078 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:10.365766 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:10.365768 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:10.366369 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:10.366371 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:10.366748 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:10.366749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:10.366800 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:10.372737 (+ 5937us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:59:10.372742 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:59:10.374586 (+ 1844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.185.126:34493: Server oryx/127.15.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":101,"threads_started":1}
W20260927 03:59:10.374775 25303 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:59:10.364647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:42957 (local address 127.15.185.126:34493)
0927 03:59:10.364709 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 03:59:10.364711 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:59:10.364729 (+ 18us) server_negotiation.cc:407] Connection header received
0927 03:59:10.364771 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:59:10.364771 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:59:10.364781 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:59:10.364841 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:59:10.365143 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:10.365641 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:10.366457 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:10.366660 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:10.366717 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:10.374691 (+ 7974us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.185.66:42957: BlockingRecv error: recv got EOF from 127.15.185.66:42957 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 27 03:59:10 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.67@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:59:10 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.67@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:59:10.477680 25305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:10.472844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34493 (local address 127.15.185.67:43637)
0927 03:59:10.473010 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 03:59:10.473024 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 03:59:10.473120 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:59:10.473302 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:59:10.473303 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:59:10.473495 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:59:10.473541 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:10.473543 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:10.474217 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:10.474218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:10.474782 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:10.474784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:10.475215 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:10.475216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:10.475282 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:10.475715 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:59:10.475720 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:59:10.477572 (+ 1852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.185.126:34493: Server oryx/127.15.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":92,"threads_started":1}
W20260927 03:59:10.477720 25303 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:59:10.472941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:43637 (local address 127.15.185.126:34493)
0927 03:59:10.473015 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0927 03:59:10.473018 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:59:10.473099 (+ 81us) server_negotiation.cc:407] Connection header received
0927 03:59:10.473206 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:59:10.473207 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:59:10.473222 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:59:10.473297 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:59:10.473621 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:10.474126 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:10.474881 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:10.475136 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:10.475204 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:10.477656 (+ 2452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.185.67:43637: BlockingRecv error: recv got EOF from 127.15.185.67:43637 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 27 03:59:11 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.65@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:59:11 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.65@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:59:11.246282 25306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:11.241688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34493 (local address 127.15.185.65:55465)
0927 03:59:11.241829 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 03:59:11.241839 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 03:59:11.241892 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:59:11.242033 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:59:11.242034 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:59:11.242193 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:59:11.242243 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:11.242245 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:11.242892 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:11.242893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:11.243480 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:11.243482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:11.243814 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:11.243814 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:11.243855 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:11.244292 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:59:11.244297 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:59:11.246222 (+ 1925us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.185.126:34493: Server oryx/127.15.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":87,"threads_started":1}
W20260927 03:59:11.246475 25307 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:59:11.241759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:55465 (local address 127.15.185.126:34493)
0927 03:59:11.241894 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 03:59:11.241896 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:59:11.241908 (+ 12us) server_negotiation.cc:407] Connection header received
0927 03:59:11.241934 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:59:11.241934 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:59:11.241945 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:59:11.242019 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:59:11.242306 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:11.242821 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:11.243552 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:11.243745 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:11.243802 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:11.246365 (+ 2563us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.185.65:55465: BlockingRecv error: recv got EOF from 127.15.185.65:55465 (error 108)
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":81,"threads_started":1}
Sep 27 03:59:11 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.66@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 03:59:11 dist-test-slave-glgz krb5kdc[24249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.185.66@KRBTEST.COM for oryx/127.15.185.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 03:59:11.380214 25308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:11.375251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34493 (local address 127.15.185.66:45813)
0927 03:59:11.375447 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 03:59:11.375459 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:59:11.375556 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:59:11.375714 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:59:11.375715 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:59:11.375895 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:59:11.375940 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:11.375941 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:11.376695 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:11.376697 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:11.377375 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:11.377377 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:11.377689 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:11.377690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:11.377732 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:11.378195 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:59:11.378199 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:59:11.380141 (+ 1942us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.185.126:34493: Server oryx/127.15.185.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":83,"threads_started":1}
W20260927 03:59:11.380323 25307 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:59:11.375327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:45813 (local address 127.15.185.126:34493)
0927 03:59:11.375421 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0927 03:59:11.375430 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:59:11.375537 (+ 107us) server_negotiation.cc:407] Connection header received
0927 03:59:11.375625 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:59:11.375626 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:59:11.375643 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:59:11.375710 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:59:11.376016 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:11.376572 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:11.377439 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:11.377626 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:11.377682 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:11.380232 (+ 2550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.185.66:45813: BlockingRecv error: recv got EOF from 127.15.185.66:45813 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 03:59:11.398113 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 24832
I20260927 03:59:11.402369 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 24984
I20260927 03:59:11.406523 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 25117
I20260927 03:59:11.410714 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 24762
2026-09-27T03:59:11Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17442 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-27T03:59:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:59:11Z Disabled control of system clock
I20260927 03:59:11.438289 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481498018407-16101-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481498018407-16101-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481498018407-16101-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:44693
--webserver_interface=127.15.185.126
--webserver_port=0
--builtin_ntp_servers=127.15.185.84:44919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.185.126:44693
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 03:59:11.518966 25312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:59:11.519222 25312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:59:11.519243 25312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:59:11.520395 25312 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:59:11.520430 25312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:59:11.520443 25312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:59:11.520457 25312 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:59:11.520468 25312 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:59:11.521736 25312 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:44919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481498018407-16101-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481498018407-16101-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.185.126:44693
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:44693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481498018407-16101-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:59:11.522008 25312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:59:11.522310 25312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 03:59:11.527390 25312 server_base.cc:1034] running on GCE node
W20260927 03:59:11.527357 25318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:59:11.527372 25320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:59:11.527350 25317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:59:11.527830 25312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:59:11.528119 25312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:59:11.529275 25312 hybrid_clock.cc:648] HybridClock initialized: now 1790481551529260 us; error 28 us; skew 500 ppm
I20260927 03:59:11.530560 25312 webserver.cc:460] Webserver started at http://127.15.185.126:43749/ using document root <none> and password file <none>
I20260927 03:59:11.530754 25312 fs_manager.cc:359] Metadata directory not provided
I20260927 03:59:11.530795 25312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:59:11.530902 25312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:59:11.532068 25312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481498018407-16101-0/minicluster-data/master-0/data/instance:
uuid: "3c7e98ac3b8d4d7fb04bea6993f7984c"
format_stamp: "Formatted at 2026-09-27 03:59:11 on dist-test-slave-glgz"
server_key: "8a64bd34d1e98526a7f03bdea0b849df"
server_key_iv: "99f6ca3c7372d82519fbd97032a2d506"
server_key_version: "encryptionkey@0"
I20260927 03:59:11.532369 25312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481498018407-16101-0/minicluster-data/master-0/wal/instance:
uuid: "3c7e98ac3b8d4d7fb04bea6993f7984c"
format_stamp: "Formatted at 2026-09-27 03:59:11 on dist-test-slave-glgz"
server_key: "8a64bd34d1e98526a7f03bdea0b849df"
server_key_iv: "99f6ca3c7372d82519fbd97032a2d506"
server_key_version: "encryptionkey@0"
I20260927 03:59:11.533563 25312 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:59:11.534204 25326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:59:11.534384 25312 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:59:11.534493 25312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481498018407-16101-0/minicluster-data/master-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481498018407-16101-0/minicluster-data/master-0/wal
uuid: "3c7e98ac3b8d4d7fb04bea6993f7984c"
format_stamp: "Formatted at 2026-09-27 03:59:11 on dist-test-slave-glgz"
server_key: "8a64bd34d1e98526a7f03bdea0b849df"
server_key_iv: "99f6ca3c7372d82519fbd97032a2d506"
server_key_version: "encryptionkey@0"
I20260927 03:59:11.534564 25312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481498018407-16101-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481498018407-16101-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481498018407-16101-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:59:11.545022 25312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:59:11.545329 25312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:59:11.545475 25312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:59:11.549152 25312 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.126:44693
I20260927 03:59:11.549196 25378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.126:44693 every 8 connection(s)
I20260927 03:59:11.549521 25312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
I20260927 03:59:11.550233 25379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:59:11.552307 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 25312
I20260927 03:59:11.552423 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481498018407-16101-0/minicluster-data/master-0/wal/instance
I20260927 03:59:11.552642 16101 external_mini_cluster.cc:1428] Setting key a04e971efbc3af0c8dda11f48a9263f5
I20260927 03:59:11.553030 25379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c7e98ac3b8d4d7fb04bea6993f7984c: Bootstrap starting.
I20260927 03:59:11.553732 25379 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c7e98ac3b8d4d7fb04bea6993f7984c: Neither blocks nor log segments found. Creating new log.
I20260927 03:59:11.554086 25379 log.cc:824] T 00000000000000000000000000000000 P 3c7e98ac3b8d4d7fb04bea6993f7984c: Log is configured to *not* fsync() on all Append() calls
I20260927 03:59:11.554917 25379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c7e98ac3b8d4d7fb04bea6993f7984c: No bootstrap required, opened a new log
I20260927 03:59:11.557307 25379 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c7e98ac3b8d4d7fb04bea6993f7984c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c7e98ac3b8d4d7fb04bea6993f7984c" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 44693 } }
I20260927 03:59:11.557440 25379 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c7e98ac3b8d4d7fb04bea6993f7984c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:59:11.557461 25379 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c7e98ac3b8d4d7fb04bea6993f7984c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c7e98ac3b8d4d7fb04bea6993f7984c, State: Initialized, Role: FOLLOWER
I20260927 03:59:11.557597 25379 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c7e98ac3b8d4d7fb04bea6993f7984c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c7e98ac3b8d4d7fb04bea6993f7984c" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 44693 } }
I20260927 03:59:11.557684 25379 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c7e98ac3b8d4d7fb04bea6993f7984c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:59:11.557725 25379 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c7e98ac3b8d4d7fb04bea6993f7984c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:59:11.557775 25379 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c7e98ac3b8d4d7fb04bea6993f7984c [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:59:11.558467 25379 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c7e98ac3b8d4d7fb04bea6993f7984c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c7e98ac3b8d4d7fb04bea6993f7984c" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 44693 } }
I20260927 03:59:11.558570 25379 leader_election.cc:302] T 00000000000000000000000000000000 P 3c7e98ac3b8d4d7fb04bea6993f7984c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c7e98ac3b8d4d7fb04bea6993f7984c; no voters:
I20260927 03:59:11.558733 25379 leader_election.cc:288] T 00000000000000000000000000000000 P 3c7e98ac3b8d4d7fb04bea6993f7984c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:59:11.558768 25384 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c7e98ac3b8d4d7fb04bea6993f7984c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:59:11.558979 25384 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c7e98ac3b8d4d7fb04bea6993f7984c [term 1 LEADER]: Becoming Leader. State: Replica: 3c7e98ac3b8d4d7fb04bea6993f7984c, State: Running, Role: LEADER
I20260927 03:59:11.559026 25379 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c7e98ac3b8d4d7fb04bea6993f7984c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:59:11.559146 25384 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c7e98ac3b8d4d7fb04bea6993f7984c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c7e98ac3b8d4d7fb04bea6993f7984c" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 44693 } }
I20260927 03:59:11.559535 25386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c7e98ac3b8d4d7fb04bea6993f7984c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c7e98ac3b8d4d7fb04bea6993f7984c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c7e98ac3b8d4d7fb04bea6993f7984c" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 44693 } } }
I20260927 03:59:11.559610 25386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c7e98ac3b8d4d7fb04bea6993f7984c [sys.catalog]: This master's current role is: LEADER
I20260927 03:59:11.559758 25386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c7e98ac3b8d4d7fb04bea6993f7984c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c7e98ac3b8d4d7fb04bea6993f7984c. Latest consensus state: current_term: 1 leader_uuid: "3c7e98ac3b8d4d7fb04bea6993f7984c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c7e98ac3b8d4d7fb04bea6993f7984c" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 44693 } } }
I20260927 03:59:11.559808 25386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c7e98ac3b8d4d7fb04bea6993f7984c [sys.catalog]: This master's current role is: LEADER
I20260927 03:59:11.560794 25382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:11.553175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40762 (local address 127.15.185.126:44693)
0927 03:59:11.553397 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 03:59:11.553408 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 03:59:11.553444 (+ 36us) server_negotiation.cc:407] Connection header received
0927 03:59:11.555158 (+ 1714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:59:11.555167 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:59:11.555243 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:59:11.555433 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:59:11.555626 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:11.557034 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:11.557992 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:11.558624 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:11.559602 (+ 978us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:11.559768 (+ 166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:59:11.559773 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:59:11.559773 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:59:11.559989 (+ 216us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:59:11.560029 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:59:11.560043 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:59:11.560390 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0927 03:59:11.560426 (+ 36us) server_negotiation.cc:299] Negotiation successful
0927 03:59:11.560584 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":82,"threads_started":1}
W20260927 03:59:11.560927 25400 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3c7e98ac3b8d4d7fb04bea6993f7984c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 03:59:11.560999 25400 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 03:59:11.561058 25391 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:59:11.561236 25391 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:59:11.562721 25391 catalog_manager.cc:1347] Generated new cluster ID: 8b27e7d903104e4e93795664c737b23a
I20260927 03:59:11.562773 25391 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:59:11.588575 25391 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:59:11.589536 25391 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:59:11.595625 25391 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c7e98ac3b8d4d7fb04bea6993f7984c: Generated new TSK 0
I20260927 03:59:11.595849 25391 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:59:11.601816 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.65:0
--local_ip_for_outbound_sockets=127.15.185.65
--webserver_interface=127.15.185.65
--webserver_port=0
--tserver_master_addrs=127.15.185.126:44693
--builtin_ntp_servers=127.15.185.84:44919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 03:59:11.700433 25403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:59:11.700659 25403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:59:11.700686 25403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:59:11.701997 25403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:59:11.702040 25403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:59:11.702067 25403 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.65
I20260927 03:59:11.703519 25403 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:44919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.185.65
--webserver_port=0
--tserver_master_addrs=127.15.185.126:44693
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.65
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:59:11.703783 25403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:59:11.704108 25403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:59:11.709324 25408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:59:11.709391 25411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:59:11.709613 25409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:59:11.709613 25403 server_base.cc:1034] running on GCE node
I20260927 03:59:11.709904 25403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:59:11.710148 25403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:59:11.711329 25403 hybrid_clock.cc:648] HybridClock initialized: now 1790481551711305 us; error 38 us; skew 500 ppm
I20260927 03:59:11.712688 25403 webserver.cc:460] Webserver started at http://127.15.185.65:42569/ using document root <none> and password file <none>
I20260927 03:59:11.712955 25403 fs_manager.cc:359] Metadata directory not provided
I20260927 03:59:11.713030 25403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:59:11.713176 25403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:59:11.714269 25403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-0/data/instance:
uuid: "1f97100814c34faf848a4cd20b0d4900"
format_stamp: "Formatted at 2026-09-27 03:59:11 on dist-test-slave-glgz"
server_key: "c232d4cad6d85e23168556f749464e7b"
server_key_iv: "fc11e50bde6e074ee0876fe5473b1ce0"
server_key_version: "encryptionkey@0"
I20260927 03:59:11.714645 25403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-0/wal/instance:
uuid: "1f97100814c34faf848a4cd20b0d4900"
format_stamp: "Formatted at 2026-09-27 03:59:11 on dist-test-slave-glgz"
server_key: "c232d4cad6d85e23168556f749464e7b"
server_key_iv: "fc11e50bde6e074ee0876fe5473b1ce0"
server_key_version: "encryptionkey@0"
I20260927 03:59:11.716053 25403 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 03:59:11.716777 25417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:59:11.716961 25403 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:59:11.717057 25403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-0/wal
uuid: "1f97100814c34faf848a4cd20b0d4900"
format_stamp: "Formatted at 2026-09-27 03:59:11 on dist-test-slave-glgz"
server_key: "c232d4cad6d85e23168556f749464e7b"
server_key_iv: "fc11e50bde6e074ee0876fe5473b1ce0"
server_key_version: "encryptionkey@0"
I20260927 03:59:11.717113 25403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:59:11.734876 25403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:59:11.735250 25403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:59:11.735368 25403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:59:11.735564 25403 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:59:11.735921 25403 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:59:11.735962 25403 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:59:11.735985 25403 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:59:11.735998 25403 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:59:11.741750 25403 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.65:35835
I20260927 03:59:11.741840 25530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.65:35835 every 8 connection(s)
I20260927 03:59:11.742183 25403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
I20260927 03:59:11.747463 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 25403
I20260927 03:59:11.747566 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-0/wal/instance
I20260927 03:59:11.747547 25382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:11.742931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:49749 (local address 127.15.185.126:44693)
0927 03:59:11.742999 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 03:59:11.743001 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:59:11.743325 (+ 324us) server_negotiation.cc:407] Connection header received
0927 03:59:11.743829 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:59:11.743830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:59:11.743843 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:59:11.743910 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:59:11.744576 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:11.745116 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:11.746145 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:11.746333 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:11.746427 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:11.747263 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:59:11.747264 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:59:11.747264 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:59:11.747269 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:59:11.747283 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:59:11.747287 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:59:11.747338 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 03:59:11.747444 (+ 106us) server_negotiation.cc:299] Negotiation successful
0927 03:59:11.747466 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 03:59:11.747793 25533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:11.742929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:44693 (local address 127.15.185.65:49749)
0927 03:59:11.743174 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0927 03:59:11.743210 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 03:59:11.743715 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:59:11.743964 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:59:11.743968 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:59:11.743976 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:59:11.744473 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:11.744477 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:11.745211 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:11.745212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:11.746051 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:11.746053 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:11.746438 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:11.746439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:11.746546 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:11.747126 (+ 580us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:59:11.747126 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:59:11.747156 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:59:11.747179 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:59:11.747349 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:59:11.747361 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:59:11.747364 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:59:11.747444 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 03:59:11.747559 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":140,"threads_started":1}
I20260927 03:59:11.747799 16101 external_mini_cluster.cc:1428] Setting key e818fee0fcf274093caf7cdd636c6451
I20260927 03:59:11.748309 25531 heartbeater.cc:347] Connected to a master server at 127.15.185.126:44693
I20260927 03:59:11.748418 25531 heartbeater.cc:464] Registering TS with master...
I20260927 03:59:11.748594 25531 heartbeater.cc:511] Master 127.15.185.126:44693 requested a full tablet report, sending...
I20260927 03:59:11.748863 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.66:0
--local_ip_for_outbound_sockets=127.15.185.66
--webserver_interface=127.15.185.66
--webserver_port=0
--tserver_master_addrs=127.15.185.126:44693
--builtin_ntp_servers=127.15.185.84:44919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 03:59:11.749133 25343 ts_manager.cc:194] Registered new tserver with Master: 1f97100814c34faf848a4cd20b0d4900 (127.15.185.65:35835)
I20260927 03:59:11.750042 25343 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.185.65:49749
W20260927 03:59:11.830691 25534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:59:11.830902 25534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:59:11.830924 25534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:59:11.832156 25534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:59:11.832201 25534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:59:11.832226 25534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.66
I20260927 03:59:11.833575 25534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:44919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.185.66
--webserver_port=0
--tserver_master_addrs=127.15.185.126:44693
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25534
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.66
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:59:11.833774 25534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:59:11.834045 25534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:59:11.839210 25542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:59:11.839212 25539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:59:11.839212 25540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:59:11.839504 25534 server_base.cc:1034] running on GCE node
I20260927 03:59:11.839751 25534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:59:11.840047 25534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:59:11.841240 25534 hybrid_clock.cc:648] HybridClock initialized: now 1790481551841229 us; error 36 us; skew 500 ppm
I20260927 03:59:11.842604 25534 webserver.cc:460] Webserver started at http://127.15.185.66:35961/ using document root <none> and password file <none>
I20260927 03:59:11.842860 25534 fs_manager.cc:359] Metadata directory not provided
I20260927 03:59:11.842928 25534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:59:11.843115 25534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:59:11.844195 25534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-1/data/instance:
uuid: "e2639db5b8214e388d962fd6107a17ab"
format_stamp: "Formatted at 2026-09-27 03:59:11 on dist-test-slave-glgz"
server_key: "d47fd3c4e44c3d25344b16e3135edace"
server_key_iv: "03009b40bdeb92f90f432a339d38a42e"
server_key_version: "encryptionkey@0"
I20260927 03:59:11.844551 25534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-1/wal/instance:
uuid: "e2639db5b8214e388d962fd6107a17ab"
format_stamp: "Formatted at 2026-09-27 03:59:11 on dist-test-slave-glgz"
server_key: "d47fd3c4e44c3d25344b16e3135edace"
server_key_iv: "03009b40bdeb92f90f432a339d38a42e"
server_key_version: "encryptionkey@0"
I20260927 03:59:11.845841 25534 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 03:59:11.846598 25548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:59:11.846872 25534 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:59:11.846994 25534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-1/wal
uuid: "e2639db5b8214e388d962fd6107a17ab"
format_stamp: "Formatted at 2026-09-27 03:59:11 on dist-test-slave-glgz"
server_key: "d47fd3c4e44c3d25344b16e3135edace"
server_key_iv: "03009b40bdeb92f90f432a339d38a42e"
server_key_version: "encryptionkey@0"
I20260927 03:59:11.847092 25534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:59:11.865841 25534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:59:11.866182 25534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:59:11.866333 25534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:59:11.866570 25534 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:59:11.866897 25534 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:59:11.866955 25534 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:59:11.866994 25534 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:59:11.867020 25534 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:59:11.872689 25534 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.66:35123
I20260927 03:59:11.872728 25661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.66:35123 every 8 connection(s)
I20260927 03:59:11.873086 25534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
I20260927 03:59:11.873986 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 25534
I20260927 03:59:11.874086 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-1/wal/instance
I20260927 03:59:11.874297 16101 external_mini_cluster.cc:1428] Setting key fe55f9eece66170f1e613cc93974f0e4
I20260927 03:59:11.875275 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.67:0
--local_ip_for_outbound_sockets=127.15.185.67
--webserver_interface=127.15.185.67
--webserver_port=0
--tserver_master_addrs=127.15.185.126:44693
--builtin_ntp_servers=127.15.185.84:44919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 03:59:11.879583 25382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:11.873813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:58923 (local address 127.15.185.126:44693)
0927 03:59:11.873882 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 03:59:11.873886 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:59:11.874143 (+ 257us) server_negotiation.cc:407] Connection header received
0927 03:59:11.874769 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:59:11.874770 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:59:11.874778 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:59:11.874839 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:59:11.875660 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:11.876460 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:11.877946 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:11.878290 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:11.878366 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:11.879255 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:59:11.879256 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:59:11.879257 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:59:11.879263 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:59:11.879281 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:59:11.879286 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:59:11.879353 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 03:59:11.879412 (+ 59us) server_negotiation.cc:299] Negotiation successful
0927 03:59:11.879440 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 03:59:11.879689 25664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:11.873805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:44693 (local address 127.15.185.66:58923)
0927 03:59:11.874002 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 03:59:11.874043 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 03:59:11.874677 (+ 634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:59:11.874927 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:59:11.874931 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:59:11.874941 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:59:11.875568 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:11.875574 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:11.876584 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:11.876585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:11.877858 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:11.877866 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:11.878438 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:11.878439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:11.878552 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:11.879125 (+ 573us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:59:11.879125 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:59:11.879154 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:59:11.879177 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:59:11.879354 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:59:11.879357 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:59:11.879360 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:59:11.879411 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 03:59:11.879496 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":93,"threads_started":1}
I20260927 03:59:11.880154 25662 heartbeater.cc:347] Connected to a master server at 127.15.185.126:44693
I20260927 03:59:11.880261 25662 heartbeater.cc:464] Registering TS with master...
I20260927 03:59:11.880487 25662 heartbeater.cc:511] Master 127.15.185.126:44693 requested a full tablet report, sending...
I20260927 03:59:11.880870 25343 ts_manager.cc:194] Registered new tserver with Master: e2639db5b8214e388d962fd6107a17ab (127.15.185.66:35123)
I20260927 03:59:11.881428 25343 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.185.66:58923
W20260927 03:59:11.958276 25665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:59:11.958489 25665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:59:11.958510 25665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:59:11.959721 25665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:59:11.959761 25665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:59:11.959786 25665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.67
I20260927 03:59:11.961175 25665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:44919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.185.67
--webserver_port=0
--tserver_master_addrs=127.15.185.126:44693
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.67
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:59:11.961367 25665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:59:11.961601 25665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:59:11.966583 25673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:59:11.966583 25670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:59:11.966678 25665 server_base.cc:1034] running on GCE node
W20260927 03:59:11.966645 25671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:59:11.967039 25665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:59:11.967319 25665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:59:11.968485 25665 hybrid_clock.cc:648] HybridClock initialized: now 1790481551968476 us; error 38 us; skew 500 ppm
I20260927 03:59:11.969767 25665 webserver.cc:460] Webserver started at http://127.15.185.67:43787/ using document root <none> and password file <none>
I20260927 03:59:11.970019 25665 fs_manager.cc:359] Metadata directory not provided
I20260927 03:59:11.970089 25665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:59:11.970235 25665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:59:11.971364 25665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-2/data/instance:
uuid: "cbe8bd3920704b21ab227a0a873e662c"
format_stamp: "Formatted at 2026-09-27 03:59:11 on dist-test-slave-glgz"
server_key: "7de9cb8eae7fc926200a083f2326c436"
server_key_iv: "8c24835025c455e1a1b56e00b48c718c"
server_key_version: "encryptionkey@0"
I20260927 03:59:11.971711 25665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-2/wal/instance:
uuid: "cbe8bd3920704b21ab227a0a873e662c"
format_stamp: "Formatted at 2026-09-27 03:59:11 on dist-test-slave-glgz"
server_key: "7de9cb8eae7fc926200a083f2326c436"
server_key_iv: "8c24835025c455e1a1b56e00b48c718c"
server_key_version: "encryptionkey@0"
I20260927 03:59:11.972966 25665 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:59:11.973680 25679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:59:11.973901 25665 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:59:11.974016 25665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-2/wal
uuid: "cbe8bd3920704b21ab227a0a873e662c"
format_stamp: "Formatted at 2026-09-27 03:59:11 on dist-test-slave-glgz"
server_key: "7de9cb8eae7fc926200a083f2326c436"
server_key_iv: "8c24835025c455e1a1b56e00b48c718c"
server_key_version: "encryptionkey@0"
I20260927 03:59:11.974090 25665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:59:11.989195 25665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:59:11.989502 25665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:59:11.989650 25665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:59:11.989879 25665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:59:11.990196 25665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:59:11.990250 25665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:59:11.990294 25665 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:59:11.990325 25665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:59:11.995922 25665 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.67:40817
I20260927 03:59:11.995971 25792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.67:40817 every 8 connection(s)
I20260927 03:59:11.996327 25665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
I20260927 03:59:12.000608 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 25665
I20260927 03:59:12.000715 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-2/wal/instance
I20260927 03:59:12.000972 16101 external_mini_cluster.cc:1428] Setting key 57c3e1a48455e30c0a202215090cee1c
I20260927 03:59:12.001729 25382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:11.997101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:53495 (local address 127.15.185.126:44693)
0927 03:59:11.997158 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 03:59:11.997160 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:59:11.997367 (+ 207us) server_negotiation.cc:407] Connection header received
0927 03:59:11.997936 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:59:11.997937 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:59:11.997946 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:59:11.997999 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:59:11.998665 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:11.999193 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:12.000126 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:12.000314 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:12.000371 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:12.001455 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:59:12.001456 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:59:12.001456 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:59:12.001461 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:59:12.001478 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:59:12.001483 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:59:12.001533 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 03:59:12.001643 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 03:59:12.001666 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 03:59:12.001999 25795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:11.997084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:44693 (local address 127.15.185.67:53495)
0927 03:59:11.997263 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 03:59:11.997297 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 03:59:11.997799 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:59:11.998056 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:59:11.998060 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:59:11.998068 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:59:11.998590 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:11.998594 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:11.999266 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:11.999267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:12.000052 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:12.000054 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:12.000435 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:12.000436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:12.000595 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:12.001262 (+ 667us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:59:12.001263 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:59:12.001303 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:59:12.001335 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:59:12.001554 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:59:12.001560 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:59:12.001563 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:59:12.001639 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 03:59:12.001747 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":85,"threads_started":1}
I20260927 03:59:12.002509 25793 heartbeater.cc:347] Connected to a master server at 127.15.185.126:44693
I20260927 03:59:12.002583 25793 heartbeater.cc:464] Registering TS with master...
I20260927 03:59:12.002722 25793 heartbeater.cc:511] Master 127.15.185.126:44693 requested a full tablet report, sending...
I20260927 03:59:12.003018 25343 ts_manager.cc:194] Registered new tserver with Master: cbe8bd3920704b21ab227a0a873e662c (127.15.185.67:40817)
I20260927 03:59:12.003377 25343 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.185.67:53495
I20260927 03:59:12.012663 16101 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 03:59:12.014444 25382 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:59:12.014009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40764 (local address 127.15.185.126:44693)
0927 03:59:12.014051 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 03:59:12.014053 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:59:12.014128 (+ 75us) server_negotiation.cc:407] Connection header received
0927 03:59:12.014187 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:59:12.014188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:59:12.014195 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:59:12.014242 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:59:12.014352 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40764: BlockingRecv error: recv got EOF from 127.0.0.1:40764 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 03:59:12.014525 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 25403
I20260927 03:59:12.019204 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 25534
I20260927 03:59:12.023487 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 25665
I20260927 03:59:12.027457 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 25312
2026-09-27T03:59:12Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (615 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-27T03:59:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:59:12Z Disabled control of system clock
I20260927 03:59:12.053673 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481498018407-16101-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481498018407-16101-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481498018407-16101-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:42041
--webserver_interface=127.15.185.126
--webserver_port=0
--builtin_ntp_servers=127.15.185.84:43751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.185.126:42041
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 03:59:12.134424 25804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:59:12.134630 25804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:59:12.134651 25804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:59:12.135830 25804 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:59:12.135870 25804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:59:12.135883 25804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:59:12.135895 25804 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:59:12.135907 25804 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:59:12.137300 25804 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:43751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481498018407-16101-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481498018407-16101-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.185.126:42041
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:42041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481498018407-16101-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:59:12.137586 25804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:59:12.137873 25804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:59:12.142701 25810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:59:12.142838 25804 server_base.cc:1034] running on GCE node
W20260927 03:59:12.142711 25809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:59:12.142711 25812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:59:12.143256 25804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:59:12.143553 25804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:59:12.144708 25804 hybrid_clock.cc:648] HybridClock initialized: now 1790481552144694 us; error 27 us; skew 500 ppm
I20260927 03:59:12.145959 25804 webserver.cc:460] Webserver started at http://127.15.185.126:43467/ using document root <none> and password file <none>
I20260927 03:59:12.146207 25804 fs_manager.cc:359] Metadata directory not provided
I20260927 03:59:12.146274 25804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:59:12.146418 25804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:59:12.147544 25804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481498018407-16101-0/minicluster-data/master-0/data/instance:
uuid: "a228e169872447f99f9d3623dff14780"
format_stamp: "Formatted at 2026-09-27 03:59:12 on dist-test-slave-glgz"
server_key: "1d2f23adf83b21f2d968558b9791c800"
server_key_iv: "750e1e218a37fcf47d25a602ab13659c"
server_key_version: "encryptionkey@0"
I20260927 03:59:12.147881 25804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481498018407-16101-0/minicluster-data/master-0/wal/instance:
uuid: "a228e169872447f99f9d3623dff14780"
format_stamp: "Formatted at 2026-09-27 03:59:12 on dist-test-slave-glgz"
server_key: "1d2f23adf83b21f2d968558b9791c800"
server_key_iv: "750e1e218a37fcf47d25a602ab13659c"
server_key_version: "encryptionkey@0"
I20260927 03:59:12.149125 25804 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 03:59:12.149873 25818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:59:12.150027 25804 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 03:59:12.150125 25804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481498018407-16101-0/minicluster-data/master-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481498018407-16101-0/minicluster-data/master-0/wal
uuid: "a228e169872447f99f9d3623dff14780"
format_stamp: "Formatted at 2026-09-27 03:59:12 on dist-test-slave-glgz"
server_key: "1d2f23adf83b21f2d968558b9791c800"
server_key_iv: "750e1e218a37fcf47d25a602ab13659c"
server_key_version: "encryptionkey@0"
I20260927 03:59:12.150194 25804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481498018407-16101-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481498018407-16101-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481498018407-16101-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:59:12.153865 25804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:59:12.154119 25804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:59:12.154258 25804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:59:12.158004 25804 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.126:42041
I20260927 03:59:12.158033 25870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.126:42041 every 8 connection(s)
I20260927 03:59:12.158357 25804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
I20260927 03:59:12.158998 25871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:59:12.161713 25871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a228e169872447f99f9d3623dff14780: Bootstrap starting.
I20260927 03:59:12.162240 25871 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a228e169872447f99f9d3623dff14780: Neither blocks nor log segments found. Creating new log.
I20260927 03:59:12.162473 25871 log.cc:824] T 00000000000000000000000000000000 P a228e169872447f99f9d3623dff14780: Log is configured to *not* fsync() on all Append() calls
I20260927 03:59:12.163152 25871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a228e169872447f99f9d3623dff14780: No bootstrap required, opened a new log
I20260927 03:59:12.164623 25871 raft_consensus.cc:348] T 00000000000000000000000000000000 P a228e169872447f99f9d3623dff14780 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a228e169872447f99f9d3623dff14780" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 42041 } }
I20260927 03:59:12.164757 25871 raft_consensus.cc:374] T 00000000000000000000000000000000 P a228e169872447f99f9d3623dff14780 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:59:12.164779 25871 raft_consensus.cc:729] T 00000000000000000000000000000000 P a228e169872447f99f9d3623dff14780 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a228e169872447f99f9d3623dff14780, State: Initialized, Role: FOLLOWER
I20260927 03:59:12.164914 25871 consensus_queue.cc:260] T 00000000000000000000000000000000 P a228e169872447f99f9d3623dff14780 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a228e169872447f99f9d3623dff14780" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 42041 } }
I20260927 03:59:12.164999 25871 raft_consensus.cc:388] T 00000000000000000000000000000000 P a228e169872447f99f9d3623dff14780 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:59:12.165021 25871 raft_consensus.cc:482] T 00000000000000000000000000000000 P a228e169872447f99f9d3623dff14780 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:59:12.165053 25871 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a228e169872447f99f9d3623dff14780 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:59:12.165522 25871 raft_consensus.cc:504] T 00000000000000000000000000000000 P a228e169872447f99f9d3623dff14780 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a228e169872447f99f9d3623dff14780" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 42041 } }
I20260927 03:59:12.165602 25871 leader_election.cc:302] T 00000000000000000000000000000000 P a228e169872447f99f9d3623dff14780 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a228e169872447f99f9d3623dff14780; no voters:
I20260927 03:59:12.165721 25871 leader_election.cc:288] T 00000000000000000000000000000000 P a228e169872447f99f9d3623dff14780 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:59:12.165788 25874 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a228e169872447f99f9d3623dff14780 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:59:12.165987 25871 sys_catalog.cc:564] T 00000000000000000000000000000000 P a228e169872447f99f9d3623dff14780 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:59:12.165988 25874 raft_consensus.cc:686] T 00000000000000000000000000000000 P a228e169872447f99f9d3623dff14780 [term 1 LEADER]: Becoming Leader. State: Replica: a228e169872447f99f9d3623dff14780, State: Running, Role: LEADER
I20260927 03:59:12.166163 25874 consensus_queue.cc:237] T 00000000000000000000000000000000 P a228e169872447f99f9d3623dff14780 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a228e169872447f99f9d3623dff14780" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 42041 } }
I20260927 03:59:12.166756 25875 sys_catalog.cc:455] T 00000000000000000000000000000000 P a228e169872447f99f9d3623dff14780 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a228e169872447f99f9d3623dff14780" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a228e169872447f99f9d3623dff14780" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 42041 } } }
I20260927 03:59:12.166777 25876 sys_catalog.cc:455] T 00000000000000000000000000000000 P a228e169872447f99f9d3623dff14780 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a228e169872447f99f9d3623dff14780. Latest consensus state: current_term: 1 leader_uuid: "a228e169872447f99f9d3623dff14780" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a228e169872447f99f9d3623dff14780" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 42041 } } }
I20260927 03:59:12.166884 25875 sys_catalog.cc:458] T 00000000000000000000000000000000 P a228e169872447f99f9d3623dff14780 [sys.catalog]: This master's current role is: LEADER
I20260927 03:59:12.166947 25876 sys_catalog.cc:458] T 00000000000000000000000000000000 P a228e169872447f99f9d3623dff14780 [sys.catalog]: This master's current role is: LEADER
I20260927 03:59:12.167286 25884 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:59:12.167825 25884 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:59:12.167986 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 25804
I20260927 03:59:12.168092 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481498018407-16101-0/minicluster-data/master-0/wal/instance
I20260927 03:59:12.168300 16101 external_mini_cluster.cc:1428] Setting key 37050987d2110bd8f3427fa1bdbbe22a
I20260927 03:59:12.169469 25884 catalog_manager.cc:1347] Generated new cluster ID: 0d994ee5e7af4aa58455625b8c2adaa1
I20260927 03:59:12.169533 25884 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:59:12.173343 25884 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:59:12.173488 25884 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:59:12.175223 25893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:12.168768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33238 (local address 127.15.185.126:42041)
0927 03:59:12.173205 (+ 4437us) server_negotiation.cc:206] Beginning negotiation
0927 03:59:12.173214 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:59:12.173248 (+ 34us) server_negotiation.cc:407] Connection header received
0927 03:59:12.173367 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:59:12.173374 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:59:12.173424 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:59:12.173583 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:59:12.174435 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:59:12.174438 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:59:12.174439 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:59:12.174443 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:59:12.174498 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:59:12.174513 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:59:12.174585 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 03:59:12.174656 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 03:59:12.174715 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4368,"thread_start_us":71,"threads_started":1}
I20260927 03:59:12.181900 25884 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a228e169872447f99f9d3623dff14780: Generated new TSK 0
I20260927 03:59:12.182099 25884 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 03:59:12.183604 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.65:0
--local_ip_for_outbound_sockets=127.15.185.65
--webserver_interface=127.15.185.65
--webserver_port=0
--tserver_master_addrs=127.15.185.126:42041
--builtin_ntp_servers=127.15.185.84:43751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 03:59:12.265085 25895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:59:12.265318 25895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:59:12.265358 25895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:59:12.266649 25895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:59:12.266709 25895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:59:12.266753 25895 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.65
I20260927 03:59:12.268266 25895 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:43751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.185.65
--webserver_port=0
--tserver_master_addrs=127.15.185.126:42041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25895
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.65
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:59:12.268534 25895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:59:12.268819 25895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:59:12.273874 25903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:59:12.274052 25895 server_base.cc:1034] running on GCE node
W20260927 03:59:12.273881 25901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:59:12.274006 25900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:59:12.274430 25895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:59:12.274686 25895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:59:12.275842 25895 hybrid_clock.cc:648] HybridClock initialized: now 1790481552275828 us; error 47 us; skew 500 ppm
I20260927 03:59:12.277175 25895 webserver.cc:460] Webserver started at http://127.15.185.65:40411/ using document root <none> and password file <none>
I20260927 03:59:12.277436 25895 fs_manager.cc:359] Metadata directory not provided
I20260927 03:59:12.277509 25895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:59:12.277652 25895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:59:12.278723 25895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-0/data/instance:
uuid: "b64f01dffc764daba6fbb5b8fdaf8a7f"
format_stamp: "Formatted at 2026-09-27 03:59:12 on dist-test-slave-glgz"
server_key: "fbaef538d53d756ebc5cbcbd6c63b9be"
server_key_iv: "d25dc3b0be09268b58704c6cda8072a5"
server_key_version: "encryptionkey@0"
I20260927 03:59:12.279170 25895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-0/wal/instance:
uuid: "b64f01dffc764daba6fbb5b8fdaf8a7f"
format_stamp: "Formatted at 2026-09-27 03:59:12 on dist-test-slave-glgz"
server_key: "fbaef538d53d756ebc5cbcbd6c63b9be"
server_key_iv: "d25dc3b0be09268b58704c6cda8072a5"
server_key_version: "encryptionkey@0"
I20260927 03:59:12.280530 25895 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:59:12.281313 25909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:59:12.281469 25895 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:59:12.281564 25895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-0/wal
uuid: "b64f01dffc764daba6fbb5b8fdaf8a7f"
format_stamp: "Formatted at 2026-09-27 03:59:12 on dist-test-slave-glgz"
server_key: "fbaef538d53d756ebc5cbcbd6c63b9be"
server_key_iv: "d25dc3b0be09268b58704c6cda8072a5"
server_key_version: "encryptionkey@0"
I20260927 03:59:12.281633 25895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:59:12.285012 25895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:59:12.285270 25895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:59:12.285401 25895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:59:12.285605 25895 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:59:12.285905 25895 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:59:12.285961 25895 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:59:12.286002 25895 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:59:12.286021 25895 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:59:12.292011 25895 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.65:46803
I20260927 03:59:12.292053 26022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.65:46803 every 8 connection(s)
I20260927 03:59:12.292368 25895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
I20260927 03:59:12.295715 25893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:12.293129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:46641 (local address 127.15.185.126:42041)
0927 03:59:12.293177 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 03:59:12.293179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:59:12.293451 (+ 272us) server_negotiation.cc:407] Connection header received
0927 03:59:12.294071 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:59:12.294072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:59:12.294078 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:59:12.294120 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:59:12.295503 (+ 1383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:59:12.295504 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:59:12.295505 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:59:12.295509 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:59:12.295523 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:59:12.295527 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:59:12.295573 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 03:59:12.295647 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 03:59:12.295661 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 03:59:12.295933 26025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:12.293129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:42041 (local address 127.15.185.65:46641)
0927 03:59:12.293301 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 03:59:12.293351 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 03:59:12.293977 (+ 626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:59:12.294196 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:59:12.294200 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:59:12.294210 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:59:12.295338 (+ 1128us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:59:12.295339 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:59:12.295378 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:59:12.295414 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:59:12.295580 (+ 166us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:59:12.295583 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:59:12.295586 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:59:12.295652 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 03:59:12.295725 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":79,"threads_started":1}
I20260927 03:59:12.296536 26023 heartbeater.cc:347] Connected to a master server at 127.15.185.126:42041
I20260927 03:59:12.296627 26023 heartbeater.cc:464] Registering TS with master...
I20260927 03:59:12.296767 26023 heartbeater.cc:511] Master 127.15.185.126:42041 requested a full tablet report, sending...
I20260927 03:59:12.297276 25835 ts_manager.cc:194] Registered new tserver with Master: b64f01dffc764daba6fbb5b8fdaf8a7f (127.15.185.65:46803)
I20260927 03:59:12.298686 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 25895
I20260927 03:59:12.298808 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-0/wal/instance
I20260927 03:59:12.299012 16101 external_mini_cluster.cc:1428] Setting key d184df12ff175f449676969746499394
I20260927 03:59:12.299960 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.66:0
--local_ip_for_outbound_sockets=127.15.185.66
--webserver_interface=127.15.185.66
--webserver_port=0
--tserver_master_addrs=127.15.185.126:42041
--builtin_ntp_servers=127.15.185.84:43751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 03:59:12.382316 26026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:59:12.382539 26026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:59:12.382561 26026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:59:12.383733 26026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:59:12.383771 26026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:59:12.383821 26026 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.66
I20260927 03:59:12.385257 26026 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:43751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.185.66
--webserver_port=0
--tserver_master_addrs=127.15.185.126:42041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.66
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:59:12.385445 26026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:59:12.385684 26026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:59:12.391507 26032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:59:12.391520 26031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:59:12.391523 26034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:59:12.392228 26026 server_base.cc:1034] running on GCE node
I20260927 03:59:12.392491 26026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:59:12.392743 26026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:59:12.393908 26026 hybrid_clock.cc:648] HybridClock initialized: now 1790481552393892 us; error 29 us; skew 500 ppm
I20260927 03:59:12.395186 26026 webserver.cc:460] Webserver started at http://127.15.185.66:42869/ using document root <none> and password file <none>
I20260927 03:59:12.395397 26026 fs_manager.cc:359] Metadata directory not provided
I20260927 03:59:12.395437 26026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:59:12.395545 26026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:59:12.396653 26026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-1/data/instance:
uuid: "df9453a2b8cf4d4aae9e86ea4d1e1e4d"
format_stamp: "Formatted at 2026-09-27 03:59:12 on dist-test-slave-glgz"
server_key: "82705a3a3ffdbc8f6c7db3865d8c8cda"
server_key_iv: "7697a0826bbd99dd52a850bd98efc33b"
server_key_version: "encryptionkey@0"
I20260927 03:59:12.396957 26026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-1/wal/instance:
uuid: "df9453a2b8cf4d4aae9e86ea4d1e1e4d"
format_stamp: "Formatted at 2026-09-27 03:59:12 on dist-test-slave-glgz"
server_key: "82705a3a3ffdbc8f6c7db3865d8c8cda"
server_key_iv: "7697a0826bbd99dd52a850bd98efc33b"
server_key_version: "encryptionkey@0"
I20260927 03:59:12.398236 26026 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260927 03:59:12.399009 26040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:59:12.399186 26026 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:59:12.399272 26026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-1/wal
uuid: "df9453a2b8cf4d4aae9e86ea4d1e1e4d"
format_stamp: "Formatted at 2026-09-27 03:59:12 on dist-test-slave-glgz"
server_key: "82705a3a3ffdbc8f6c7db3865d8c8cda"
server_key_iv: "7697a0826bbd99dd52a850bd98efc33b"
server_key_version: "encryptionkey@0"
I20260927 03:59:12.399338 26026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:59:12.402716 26026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:59:12.402935 26026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:59:12.403031 26026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:59:12.403247 26026 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:59:12.403496 26026 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:59:12.403527 26026 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:59:12.403546 26026 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:59:12.403563 26026 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:59:12.409163 26026 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.66:36985
I20260927 03:59:12.409215 26153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.66:36985 every 8 connection(s)
I20260927 03:59:12.409516 26026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
I20260927 03:59:12.412521 25893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:12.410225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:33699 (local address 127.15.185.126:42041)
0927 03:59:12.410284 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 03:59:12.410286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:59:12.410516 (+ 230us) server_negotiation.cc:407] Connection header received
0927 03:59:12.411026 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:59:12.411027 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:59:12.411033 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:59:12.411126 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:59:12.412256 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:59:12.412258 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:59:12.412258 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:59:12.412263 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:59:12.412280 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:59:12.412284 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:59:12.412345 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 03:59:12.412429 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 03:59:12.412450 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 03:59:12.412716 26156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:12.410232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:42041 (local address 127.15.185.66:33699)
0927 03:59:12.410401 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 03:59:12.410434 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 03:59:12.410904 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:59:12.411196 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:59:12.411205 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:59:12.411213 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:59:12.412100 (+ 887us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:59:12.412101 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:59:12.412129 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:59:12.412157 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:59:12.412341 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:59:12.412346 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:59:12.412350 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:59:12.412441 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 03:59:12.412503 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":71,"threads_started":1}
I20260927 03:59:12.413164 26154 heartbeater.cc:347] Connected to a master server at 127.15.185.126:42041
I20260927 03:59:12.413257 26154 heartbeater.cc:464] Registering TS with master...
I20260927 03:59:12.413412 26154 heartbeater.cc:511] Master 127.15.185.126:42041 requested a full tablet report, sending...
I20260927 03:59:12.413796 25835 ts_manager.cc:194] Registered new tserver with Master: df9453a2b8cf4d4aae9e86ea4d1e1e4d (127.15.185.66:36985)
I20260927 03:59:12.414388 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 26026
I20260927 03:59:12.414520 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-1/wal/instance
I20260927 03:59:12.414716 16101 external_mini_cluster.cc:1428] Setting key a85a701015d796a5465799ac77a6a6f0
I20260927 03:59:12.415580 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.67:0
--local_ip_for_outbound_sockets=127.15.185.67
--webserver_interface=127.15.185.67
--webserver_port=0
--tserver_master_addrs=127.15.185.126:42041
--builtin_ntp_servers=127.15.185.84:43751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 03:59:12.495302 26157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:59:12.495513 26157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:59:12.495536 26157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:59:12.496690 26157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:59:12.496728 26157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:59:12.496754 26157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.67
I20260927 03:59:12.498090 26157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:43751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.185.67
--webserver_port=0
--tserver_master_addrs=127.15.185.126:42041
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.67
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:59:12.498328 26157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:59:12.498633 26157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:59:12.503723 26165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:59:12.503767 26163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:59:12.503993 26157 server_base.cc:1034] running on GCE node
W20260927 03:59:12.504019 26162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:59:12.504231 26157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:59:12.504432 26157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:59:12.505571 26157 hybrid_clock.cc:648] HybridClock initialized: now 1790481552505559 us; error 27 us; skew 500 ppm
I20260927 03:59:12.506768 26157 webserver.cc:460] Webserver started at http://127.15.185.67:41205/ using document root <none> and password file <none>
I20260927 03:59:12.506968 26157 fs_manager.cc:359] Metadata directory not provided
I20260927 03:59:12.507010 26157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:59:12.507146 26157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:59:12.508170 26157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-2/data/instance:
uuid: "b23d139d1733416ca81ad4d8009a8d2c"
format_stamp: "Formatted at 2026-09-27 03:59:12 on dist-test-slave-glgz"
server_key: "188c3cf214dd6c1f036e6a31e2d32712"
server_key_iv: "4a6963be07f0b056b8b637772384af90"
server_key_version: "encryptionkey@0"
I20260927 03:59:12.508486 26157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-2/wal/instance:
uuid: "b23d139d1733416ca81ad4d8009a8d2c"
format_stamp: "Formatted at 2026-09-27 03:59:12 on dist-test-slave-glgz"
server_key: "188c3cf214dd6c1f036e6a31e2d32712"
server_key_iv: "4a6963be07f0b056b8b637772384af90"
server_key_version: "encryptionkey@0"
I20260927 03:59:12.509627 26157 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 03:59:12.510253 26171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:59:12.510414 26157 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:59:12.510519 26157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-2/wal
uuid: "b23d139d1733416ca81ad4d8009a8d2c"
format_stamp: "Formatted at 2026-09-27 03:59:12 on dist-test-slave-glgz"
server_key: "188c3cf214dd6c1f036e6a31e2d32712"
server_key_iv: "4a6963be07f0b056b8b637772384af90"
server_key_version: "encryptionkey@0"
I20260927 03:59:12.510591 26157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:59:12.513922 26157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:59:12.514170 26157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:59:12.514303 26157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:59:12.514508 26157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:59:12.514796 26157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:59:12.514844 26157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:59:12.514881 26157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:59:12.514909 26157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:59:12.520586 26157 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.67:35345
I20260927 03:59:12.520632 26284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.67:35345 every 8 connection(s)
I20260927 03:59:12.520926 26157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
I20260927 03:59:12.523929 25893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:12.521594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:49843 (local address 127.15.185.126:42041)
0927 03:59:12.521651 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 03:59:12.521653 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:59:12.521921 (+ 268us) server_negotiation.cc:407] Connection header received
0927 03:59:12.522432 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:59:12.522433 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:59:12.522439 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:59:12.522497 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:59:12.523678 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:59:12.523680 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:59:12.523680 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 03:59:12.523685 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:59:12.523700 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 03:59:12.523705 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:59:12.523756 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 03:59:12.523838 (+ 82us) server_negotiation.cc:299] Negotiation successful
0927 03:59:12.523854 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 03:59:12.524142 26287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:12.521592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:42041 (local address 127.15.185.67:49843)
0927 03:59:12.521755 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 03:59:12.521789 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 03:59:12.522324 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:59:12.522576 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:59:12.522579 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:59:12.522588 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:59:12.523526 (+ 938us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 03:59:12.523526 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:59:12.523555 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 03:59:12.523585 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:59:12.523761 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:59:12.523765 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:59:12.523768 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 03:59:12.523849 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 03:59:12.523912 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":72,"threads_started":1}
I20260927 03:59:12.524650 26285 heartbeater.cc:347] Connected to a master server at 127.15.185.126:42041
I20260927 03:59:12.524748 26285 heartbeater.cc:464] Registering TS with master...
I20260927 03:59:12.524933 26285 heartbeater.cc:511] Master 127.15.185.126:42041 requested a full tablet report, sending...
I20260927 03:59:12.525271 25835 ts_manager.cc:194] Registered new tserver with Master: b23d139d1733416ca81ad4d8009a8d2c (127.15.185.67:35345)
I20260927 03:59:12.529860 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 26157
I20260927 03:59:12.529968 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481498018407-16101-0/minicluster-data/ts-2/wal/instance
I20260927 03:59:12.530157 16101 external_mini_cluster.cc:1428] Setting key 32a616d83ef746352944401bc8f90d38
I20260927 03:59:12.531020 16101 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 03:59:12.532855 25893 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:59:12.532332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33250 (local address 127.15.185.126:42041)
0927 03:59:12.532396 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 03:59:12.532399 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:59:12.532560 (+ 161us) server_negotiation.cc:407] Connection header received
0927 03:59:12.532599 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:59:12.532600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:59:12.532605 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:59:12.532660 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:59:12.532798 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33250: BlockingRecv error: recv got EOF from 127.0.0.1:33250 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 03:59:12.532919 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 25895
I20260927 03:59:12.538024 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 26026
I20260927 03:59:12.542425 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 26157
I20260927 03:59:12.546777 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 25804
2026-09-27T03:59:12Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (519 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:59:12 dist-test-slave-glgz krb5kdc[26295](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:59:12 dist-test-slave-glgz krb5kdc[26295](info): set up 2 sockets
Sep 27 03:59:12 dist-test-slave-glgz krb5kdc[26295](info): commencing operation
krb5kdc: starting...
W20260927 03:59:13.588433 16101 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Sep 27 03:59:13 dist-test-slave-glgz krb5kdc[26295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:59:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:59:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.185.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab.
Entry for principal kudu/127.15.185.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab.
I20260927 03:59:13.706138 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:34747
--webserver_interface=127.15.185.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:33511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.185.126:34747
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:59:13.786816 26308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:59:13.787094 26308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:59:13.787132 26308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:59:13.788337 26308 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:59:13.788412 26308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:59:13.788435 26308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:59:13.788456 26308 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:59:13.788484 26308 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:59:13.789983 26308 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:33511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.185.126:34747
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:34747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:59:13.790268 26308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:59:13.790508 26308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:59:13.795360 26313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:59:13.795492 26308 server_base.cc:1034] running on GCE node
W20260927 03:59:13.795337 26316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:59:13.795332 26314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:59:13.795854 26308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:59:13.796172 26308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:59:13.797322 26308 hybrid_clock.cc:648] HybridClock initialized: now 1790481553797304 us; error 37 us; skew 500 ppm
Sep 27 03:59:13 dist-test-slave-glgz krb5kdc[26295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481553, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:59:13.799856 26308 init.cc:377] Logged in from keytab as kudu/127.15.185.126@KRBTEST.COM (short username kudu)
I20260927 03:59:13.800295 26308 webserver.cc:460] Webserver started at http://127.15.185.126:40969/ using document root <none> and password file <none>
I20260927 03:59:13.800541 26308 fs_manager.cc:359] Metadata directory not provided
I20260927 03:59:13.800587 26308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:59:13.800678 26308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:59:13.801702 26308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/minicluster-data/master-0/data/instance:
uuid: "cce83b5c7e1f44e3ba08932330d49b8c"
format_stamp: "Formatted at 2026-09-27 03:59:13 on dist-test-slave-glgz"
server_key: "4765e607e2d0b7322e4e0c0328f87e2a"
server_key_iv: "ebef3d1a2c997ed496c334a83642ed6d"
server_key_version: "encryptionkey@0"
I20260927 03:59:13.802232 26308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/minicluster-data/master-0/wal/instance:
uuid: "cce83b5c7e1f44e3ba08932330d49b8c"
format_stamp: "Formatted at 2026-09-27 03:59:13 on dist-test-slave-glgz"
server_key: "4765e607e2d0b7322e4e0c0328f87e2a"
server_key_iv: "ebef3d1a2c997ed496c334a83642ed6d"
server_key_version: "encryptionkey@0"
I20260927 03:59:13.803710 26308 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260927 03:59:13.804562 26323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:59:13.804744 26308 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:59:13.804841 26308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/minicluster-data/master-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/minicluster-data/master-0/wal
uuid: "cce83b5c7e1f44e3ba08932330d49b8c"
format_stamp: "Formatted at 2026-09-27 03:59:13 on dist-test-slave-glgz"
server_key: "4765e607e2d0b7322e4e0c0328f87e2a"
server_key_iv: "ebef3d1a2c997ed496c334a83642ed6d"
server_key_version: "encryptionkey@0"
I20260927 03:59:13.804949 26308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:59:13.843669 26308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:59:13.846032 26308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:59:13.846148 26308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:59:13.850119 26308 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.126:34747
I20260927 03:59:13.850183 26375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.126:34747 every 8 connection(s)
I20260927 03:59:13.850512 26308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
I20260927 03:59:13.851205 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 26308
I20260927 03:59:13.851286 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/minicluster-data/master-0/wal/instance
I20260927 03:59:13.851502 16101 external_mini_cluster.cc:1428] Setting key 6d4fcc2dc8fa9d180464262902d25400
I20260927 03:59:13.851649 26376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:59:13.855000 26376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cce83b5c7e1f44e3ba08932330d49b8c: Bootstrap starting.
I20260927 03:59:13.855621 26376 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cce83b5c7e1f44e3ba08932330d49b8c: Neither blocks nor log segments found. Creating new log.
I20260927 03:59:13.855916 26376 log.cc:824] T 00000000000000000000000000000000 P cce83b5c7e1f44e3ba08932330d49b8c: Log is configured to *not* fsync() on all Append() calls
I20260927 03:59:13.856578 26376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cce83b5c7e1f44e3ba08932330d49b8c: No bootstrap required, opened a new log
Sep 27 03:59:13 dist-test-slave-glgz krb5kdc[26295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 03:59:13.858969 26376 raft_consensus.cc:348] T 00000000000000000000000000000000 P cce83b5c7e1f44e3ba08932330d49b8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cce83b5c7e1f44e3ba08932330d49b8c" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 34747 } }
I20260927 03:59:13.859164 26376 raft_consensus.cc:374] T 00000000000000000000000000000000 P cce83b5c7e1f44e3ba08932330d49b8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:59:13.859210 26376 raft_consensus.cc:729] T 00000000000000000000000000000000 P cce83b5c7e1f44e3ba08932330d49b8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cce83b5c7e1f44e3ba08932330d49b8c, State: Initialized, Role: FOLLOWER
I20260927 03:59:13.859341 26376 consensus_queue.cc:260] T 00000000000000000000000000000000 P cce83b5c7e1f44e3ba08932330d49b8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cce83b5c7e1f44e3ba08932330d49b8c" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 34747 } }
I20260927 03:59:13.859438 26376 raft_consensus.cc:388] T 00000000000000000000000000000000 P cce83b5c7e1f44e3ba08932330d49b8c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:59:13.859489 26376 raft_consensus.cc:482] T 00000000000000000000000000000000 P cce83b5c7e1f44e3ba08932330d49b8c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:59:13.859539 26376 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cce83b5c7e1f44e3ba08932330d49b8c [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:59:13.860155 26376 raft_consensus.cc:504] T 00000000000000000000000000000000 P cce83b5c7e1f44e3ba08932330d49b8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cce83b5c7e1f44e3ba08932330d49b8c" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 34747 } }
I20260927 03:59:13.860271 26376 leader_election.cc:302] T 00000000000000000000000000000000 P cce83b5c7e1f44e3ba08932330d49b8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cce83b5c7e1f44e3ba08932330d49b8c; no voters:
I20260927 03:59:13.860467 26376 leader_election.cc:288] T 00000000000000000000000000000000 P cce83b5c7e1f44e3ba08932330d49b8c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:59:13.860499 26381 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cce83b5c7e1f44e3ba08932330d49b8c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:59:13.860620 26381 raft_consensus.cc:686] T 00000000000000000000000000000000 P cce83b5c7e1f44e3ba08932330d49b8c [term 1 LEADER]: Becoming Leader. State: Replica: cce83b5c7e1f44e3ba08932330d49b8c, State: Running, Role: LEADER
I20260927 03:59:13.860730 26376 sys_catalog.cc:564] T 00000000000000000000000000000000 P cce83b5c7e1f44e3ba08932330d49b8c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:59:13.860762 26381 consensus_queue.cc:237] T 00000000000000000000000000000000 P cce83b5c7e1f44e3ba08932330d49b8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cce83b5c7e1f44e3ba08932330d49b8c" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 34747 } }
I20260927 03:59:13.861107 26381 sys_catalog.cc:455] T 00000000000000000000000000000000 P cce83b5c7e1f44e3ba08932330d49b8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader cce83b5c7e1f44e3ba08932330d49b8c. Latest consensus state: current_term: 1 leader_uuid: "cce83b5c7e1f44e3ba08932330d49b8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cce83b5c7e1f44e3ba08932330d49b8c" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 34747 } } }
I20260927 03:59:13.861279 26381 sys_catalog.cc:458] T 00000000000000000000000000000000 P cce83b5c7e1f44e3ba08932330d49b8c [sys.catalog]: This master's current role is: LEADER
I20260927 03:59:13.861119 26383 sys_catalog.cc:455] T 00000000000000000000000000000000 P cce83b5c7e1f44e3ba08932330d49b8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cce83b5c7e1f44e3ba08932330d49b8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cce83b5c7e1f44e3ba08932330d49b8c" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 34747 } } }
I20260927 03:59:13.861405 26383 sys_catalog.cc:458] T 00000000000000000000000000000000 P cce83b5c7e1f44e3ba08932330d49b8c [sys.catalog]: This master's current role is: LEADER
I20260927 03:59:13.862084 26390 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:59:13.862738 26390 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:59:13.862799 26379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:13.851990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32826 (local address 127.15.185.126:34747)
0927 03:59:13.852169 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 03:59:13.852178 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 03:59:13.852212 (+ 34us) server_negotiation.cc:407] Connection header received
0927 03:59:13.852834 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:59:13.852843 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:59:13.852905 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:59:13.853059 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:59:13.853613 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:13.854970 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:13.855782 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:13.856236 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:13.856368 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:13.858838 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:59:13.858842 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:59:13.858842 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:59:13.858852 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:59:13.860650 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:59:13.861076 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:59:13.861078 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:59:13.861079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:59:13.861184 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:59:13.861396 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:59:13.861396 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:59:13.861397 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:59:13.861767 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:59:13.862375 (+ 608us) server_negotiation.cc:1035] Waiting for connection context
0927 03:59:13.862455 (+ 80us) server_negotiation.cc:299] Negotiation successful
0927 03:59:13.862601 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":79,"threads_started":1}
I20260927 03:59:13.864328 26390 catalog_manager.cc:1347] Generated new cluster ID: cdec213fc7be4af1bf472fc47e789bf6
I20260927 03:59:13.864377 26390 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:59:13.870445 26390 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:59:13.870959 26390 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:59:13.885902 26390 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cce83b5c7e1f44e3ba08932330d49b8c: Generated new TSK 0
I20260927 03:59:13.886137 26390 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.185.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab.
Entry for principal kudu/127.15.185.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab.
I20260927 03:59:13.914391 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.65:0
--local_ip_for_outbound_sockets=127.15.185.65
--webserver_interface=127.15.185.65
--webserver_port=0
--tserver_master_addrs=127.15.185.126:34747
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab
--principal=kudu/127.15.185.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:33511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:59:13.996279 26402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:59:13.996503 26402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:59:13.996528 26402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:59:13.997658 26402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:59:13.997696 26402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:59:13.997722 26402 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.65
I20260927 03:59:13.999217 26402 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:33511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab
--principal=kudu/127.15.185.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.185.65
--webserver_port=0
--tserver_master_addrs=127.15.185.126:34747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26402
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.65
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:59:13.999413 26402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:59:13.999596 26402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:59:14.004508 26410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:59:14.004508 26408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:59:14.004545 26407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:59:14.004801 26402 server_base.cc:1034] running on GCE node
I20260927 03:59:14.005050 26402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:59:14.005256 26402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:59:14.006418 26402 hybrid_clock.cc:648] HybridClock initialized: now 1790481554006405 us; error 40 us; skew 500 ppm
Sep 27 03:59:14 dist-test-slave-glgz krb5kdc[26295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481554, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:59:14.009126 26402 init.cc:377] Logged in from keytab as kudu/127.15.185.65@KRBTEST.COM (short username kudu)
I20260927 03:59:14.009572 26402 webserver.cc:460] Webserver started at http://127.15.185.65:41813/ using document root <none> and password file <none>
I20260927 03:59:14.009761 26402 fs_manager.cc:359] Metadata directory not provided
I20260927 03:59:14.009801 26402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:59:14.009891 26402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:59:14.010931 26402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/minicluster-data/ts-0/data/instance:
uuid: "1557db66f9a8494eaa4f6a8567ffb2a5"
format_stamp: "Formatted at 2026-09-27 03:59:14 on dist-test-slave-glgz"
server_key: "da9c2b93ccd8444537a51ae2fe831183"
server_key_iv: "6b3f298d07de241e56164e5c6cc7179d"
server_key_version: "encryptionkey@0"
I20260927 03:59:14.011304 26402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/minicluster-data/ts-0/wal/instance:
uuid: "1557db66f9a8494eaa4f6a8567ffb2a5"
format_stamp: "Formatted at 2026-09-27 03:59:14 on dist-test-slave-glgz"
server_key: "da9c2b93ccd8444537a51ae2fe831183"
server_key_iv: "6b3f298d07de241e56164e5c6cc7179d"
server_key_version: "encryptionkey@0"
I20260927 03:59:14.012542 26402 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:59:14.013226 26417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:59:14.013370 26402 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:59:14.013469 26402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/minicluster-data/ts-0/wal
uuid: "1557db66f9a8494eaa4f6a8567ffb2a5"
format_stamp: "Formatted at 2026-09-27 03:59:14 on dist-test-slave-glgz"
server_key: "da9c2b93ccd8444537a51ae2fe831183"
server_key_iv: "6b3f298d07de241e56164e5c6cc7179d"
server_key_version: "encryptionkey@0"
I20260927 03:59:14.013525 26402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:59:14.028070 26402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:59:14.030263 26402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:59:14.030372 26402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:59:14.030557 26402 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:59:14.030805 26402 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:59:14.030834 26402 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:59:14.030857 26402 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:59:14.030870 26402 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:59:14.036442 26402 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.65:42897
I20260927 03:59:14.036543 26530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.65:42897 every 8 connection(s)
I20260927 03:59:14.036837 26402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
I20260927 03:59:14.039539 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 26402
I20260927 03:59:14.039654 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/minicluster-data/ts-0/wal/instance
I20260927 03:59:14.039855 16101 external_mini_cluster.cc:1428] Setting key f0b601b9e6f26e6f1d8f30c8d4a93ba9
Sep 27 03:59:14 dist-test-slave-glgz krb5kdc[26295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481554, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.65@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 03:59:14.046694 26379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:14.037595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:48801 (local address 127.15.185.126:34747)
0927 03:59:14.037673 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0927 03:59:14.037676 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:59:14.037820 (+ 144us) server_negotiation.cc:407] Connection header received
0927 03:59:14.038304 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:59:14.038305 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:59:14.038315 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:59:14.038370 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:59:14.039308 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:14.039863 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:14.041084 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:14.041306 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:14.041388 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:14.043737 (+ 2349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:59:14.043738 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:59:14.043738 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:59:14.043745 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:59:14.045189 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:59:14.045704 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:59:14.045704 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:59:14.045705 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:59:14.045757 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:59:14.046046 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:59:14.046046 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:59:14.046046 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:59:14.046149 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:59:14.046197 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 03:59:14.046546 (+ 349us) server_negotiation.cc:299] Negotiation successful
0927 03:59:14.046626 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 03:59:14.046967 26533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:14.037511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34747 (local address 127.15.185.65:48801)
0927 03:59:14.037665 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 03:59:14.037702 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 03:59:14.038220 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:59:14.038437 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:59:14.038440 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:59:14.039013 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:59:14.039230 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:14.039235 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:14.039965 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:14.039966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:14.040994 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:14.040996 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:14.041397 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:14.041398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:14.041537 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:14.042058 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:59:14.042066 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:59:14.043608 (+ 1542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:59:14.045275 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:59:14.045284 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:59:14.045287 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:59:14.045624 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:59:14.045824 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:59:14.045825 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:59:14.045826 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:59:14.045976 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:59:14.046212 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:59:14.046218 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:59:14.046399 (+ 181us) client_negotiation.cc:769] Sending connection context
0927 03:59:14.046520 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 03:59:14.046695 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":61,"threads_started":1}
I20260927 03:59:14.047552 26531 heartbeater.cc:347] Connected to a master server at 127.15.185.126:34747
I20260927 03:59:14.047658 26531 heartbeater.cc:464] Registering TS with master...
I20260927 03:59:14.047868 26531 heartbeater.cc:511] Master 127.15.185.126:34747 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.15.185.66@KRBTEST.COM; defaulting to no policy
I20260927 03:59:14.048379 26340 ts_manager.cc:194] Registered new tserver with Master: 1557db66f9a8494eaa4f6a8567ffb2a5 (127.15.185.65:42897)
I20260927 03:59:14.049054 26340 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.185.65@KRBTEST.COM'} at 127.15.185.65:48801
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab.
Entry for principal kudu/127.15.185.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab.
I20260927 03:59:14.065495 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.66:0
--local_ip_for_outbound_sockets=127.15.185.66
--webserver_interface=127.15.185.66
--webserver_port=0
--tserver_master_addrs=127.15.185.126:34747
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab
--principal=kudu/127.15.185.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:33511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:59:14.147670 26536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:59:14.147883 26536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:59:14.147907 26536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:59:14.149080 26536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:59:14.149120 26536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:59:14.149173 26536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.66
I20260927 03:59:14.150642 26536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:33511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab
--principal=kudu/127.15.185.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.185.66
--webserver_port=0
--tserver_master_addrs=127.15.185.126:34747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26536
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.66
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:59:14.150895 26536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:59:14.151181 26536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:59:14.156209 26541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:59:14.156209 26544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:59:14.156224 26542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:59:14.156414 26536 server_base.cc:1034] running on GCE node
I20260927 03:59:14.156729 26536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:59:14.156980 26536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:59:14.158130 26536 hybrid_clock.cc:648] HybridClock initialized: now 1790481554158122 us; error 31 us; skew 500 ppm
Sep 27 03:59:14 dist-test-slave-glgz krb5kdc[26295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481554, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:59:14.160769 26536 init.cc:377] Logged in from keytab as kudu/127.15.185.66@KRBTEST.COM (short username kudu)
I20260927 03:59:14.161197 26536 webserver.cc:460] Webserver started at http://127.15.185.66:40385/ using document root <none> and password file <none>
I20260927 03:59:14.161432 26536 fs_manager.cc:359] Metadata directory not provided
I20260927 03:59:14.161496 26536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:59:14.161587 26536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:59:14.162690 26536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/minicluster-data/ts-1/data/instance:
uuid: "52728bcd4f874803a2ecb114b3b19455"
format_stamp: "Formatted at 2026-09-27 03:59:14 on dist-test-slave-glgz"
server_key: "daae2409243f7268110c6be6c051a7f1"
server_key_iv: "122410e10c949bd19e4c1d46b971c135"
server_key_version: "encryptionkey@0"
I20260927 03:59:14.162998 26536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/minicluster-data/ts-1/wal/instance:
uuid: "52728bcd4f874803a2ecb114b3b19455"
format_stamp: "Formatted at 2026-09-27 03:59:14 on dist-test-slave-glgz"
server_key: "daae2409243f7268110c6be6c051a7f1"
server_key_iv: "122410e10c949bd19e4c1d46b971c135"
server_key_version: "encryptionkey@0"
I20260927 03:59:14.164278 26536 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:59:14.164960 26551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:59:14.165107 26536 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:59:14.165187 26536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/minicluster-data/ts-1/wal
uuid: "52728bcd4f874803a2ecb114b3b19455"
format_stamp: "Formatted at 2026-09-27 03:59:14 on dist-test-slave-glgz"
server_key: "daae2409243f7268110c6be6c051a7f1"
server_key_iv: "122410e10c949bd19e4c1d46b971c135"
server_key_version: "encryptionkey@0"
I20260927 03:59:14.165241 26536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:59:14.174882 26536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:59:14.176743 26536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:59:14.176851 26536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:59:14.177023 26536 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:59:14.177347 26536 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:59:14.177381 26536 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:59:14.177434 26536 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:59:14.177467 26536 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:59:14.183023 26536 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.66:39567
I20260927 03:59:14.183128 26664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.66:39567 every 8 connection(s)
I20260927 03:59:14.183450 26536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
Sep 27 03:59:14 dist-test-slave-glgz krb5kdc[26295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481554, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.66@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 03:59:14.190114 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 26536
I20260927 03:59:14.190231 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/minicluster-data/ts-1/wal/instance
I20260927 03:59:14.190446 16101 external_mini_cluster.cc:1428] Setting key f0840e230e1558423b2641ccea7b8ddb
I20260927 03:59:14.192873 26379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:14.184209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:55251 (local address 127.15.185.126:34747)
0927 03:59:14.184283 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0927 03:59:14.184286 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 03:59:14.184470 (+ 184us) server_negotiation.cc:407] Connection header received
0927 03:59:14.185016 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:59:14.185018 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:59:14.185031 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:59:14.185085 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:59:14.186028 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:14.186536 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:14.187574 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:14.187766 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:14.187862 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:14.189959 (+ 2097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:59:14.189961 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:59:14.189961 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:59:14.189969 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:59:14.191524 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:59:14.191939 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:59:14.191940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:59:14.191941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:59:14.191999 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:59:14.192246 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:59:14.192247 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:59:14.192247 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:59:14.192362 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:59:14.192433 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 03:59:14.192698 (+ 265us) server_negotiation.cc:299] Negotiation successful
0927 03:59:14.192777 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 03:59:14.193095 26667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:14.184189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34747 (local address 127.15.185.66:55251)
0927 03:59:14.184344 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 03:59:14.184380 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 03:59:14.184881 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:59:14.185159 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:59:14.185162 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:59:14.185726 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:59:14.185919 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:14.185923 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:14.186627 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:14.186628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:14.187489 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:14.187491 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:14.187867 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:14.187868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:14.188002 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:14.188541 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:59:14.188549 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:59:14.189866 (+ 1317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:59:14.191620 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:59:14.191627 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:59:14.191629 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:59:14.191854 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:59:14.192063 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:59:14.192064 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:59:14.192064 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:59:14.192181 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:59:14.192424 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:59:14.192430 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:59:14.192573 (+ 143us) client_negotiation.cc:769] Sending connection context
0927 03:59:14.192680 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 03:59:14.192841 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":73,"threads_started":1}
I20260927 03:59:14.193619 26665 heartbeater.cc:347] Connected to a master server at 127.15.185.126:34747
I20260927 03:59:14.193723 26665 heartbeater.cc:464] Registering TS with master...
I20260927 03:59:14.193928 26665 heartbeater.cc:511] Master 127.15.185.126:34747 requested a full tablet report, sending...
I20260927 03:59:14.194324 26340 ts_manager.cc:194] Registered new tserver with Master: 52728bcd4f874803a2ecb114b3b19455 (127.15.185.66:39567)
I20260927 03:59:14.194756 26340 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.185.66@KRBTEST.COM'} at 127.15.185.66:55251
WARNING: no policy specified for kudu/127.15.185.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab.
Entry for principal kudu/127.15.185.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab.
I20260927 03:59:14.217701 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.67:0
--local_ip_for_outbound_sockets=127.15.185.67
--webserver_interface=127.15.185.67
--webserver_port=0
--tserver_master_addrs=127.15.185.126:34747
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab
--principal=kudu/127.15.185.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:33511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:59:14.298605 26670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:59:14.298873 26670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:59:14.298913 26670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:59:14.300251 26670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:59:14.300324 26670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:59:14.300370 26670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.67
I20260927 03:59:14.301806 26670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:33511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab
--principal=kudu/127.15.185.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.185.67
--webserver_port=0
--tserver_master_addrs=127.15.185.126:34747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.67
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:59:14.302081 26670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:59:14.302311 26670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:59:14.308087 26675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:59:14.308205 26670 server_base.cc:1034] running on GCE node
W20260927 03:59:14.308120 26678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:59:14.308065 26676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:59:14.308548 26670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:59:14.308815 26670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:59:14.309991 26670 hybrid_clock.cc:648] HybridClock initialized: now 1790481554309988 us; error 34 us; skew 500 ppm
Sep 27 03:59:14 dist-test-slave-glgz krb5kdc[26295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481554, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:59:14.312556 26670 init.cc:377] Logged in from keytab as kudu/127.15.185.67@KRBTEST.COM (short username kudu)
I20260927 03:59:14.313019 26670 webserver.cc:460] Webserver started at http://127.15.185.67:39485/ using document root <none> and password file <none>
I20260927 03:59:14.313241 26670 fs_manager.cc:359] Metadata directory not provided
I20260927 03:59:14.313303 26670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:59:14.313445 26670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 03:59:14.314498 26670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/minicluster-data/ts-2/data/instance:
uuid: "65bb143dae144cd581d2109cf741ba88"
format_stamp: "Formatted at 2026-09-27 03:59:14 on dist-test-slave-glgz"
server_key: "a9cf70f7a007e642c4bcaf60131d1777"
server_key_iv: "b46f2b40c3b870932a81b659192aa174"
server_key_version: "encryptionkey@0"
I20260927 03:59:14.314848 26670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/minicluster-data/ts-2/wal/instance:
uuid: "65bb143dae144cd581d2109cf741ba88"
format_stamp: "Formatted at 2026-09-27 03:59:14 on dist-test-slave-glgz"
server_key: "a9cf70f7a007e642c4bcaf60131d1777"
server_key_iv: "b46f2b40c3b870932a81b659192aa174"
server_key_version: "encryptionkey@0"
I20260927 03:59:14.316213 26670 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:59:14.317016 26685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:59:14.317199 26670 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 03:59:14.317293 26670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/minicluster-data/ts-2/wal
uuid: "65bb143dae144cd581d2109cf741ba88"
format_stamp: "Formatted at 2026-09-27 03:59:14 on dist-test-slave-glgz"
server_key: "a9cf70f7a007e642c4bcaf60131d1777"
server_key_iv: "b46f2b40c3b870932a81b659192aa174"
server_key_version: "encryptionkey@0"
I20260927 03:59:14.317373 26670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:59:14.331811 26670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:59:14.333729 26670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:59:14.333907 26670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:59:14.334146 26670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 03:59:14.334458 26670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 03:59:14.334509 26670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 03:59:14.334550 26670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 03:59:14.334579 26670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 03:59:14.340332 26670 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.67:43413
I20260927 03:59:14.340399 26798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.67:43413 every 8 connection(s)
I20260927 03:59:14.340751 26670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
I20260927 03:59:14.342514 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 26670
I20260927 03:59:14.342645 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/minicluster-data/ts-2/wal/instance
I20260927 03:59:14.342873 16101 external_mini_cluster.cc:1428] Setting key 83e55add8a2dcc68ee96854a39373d5d
Sep 27 03:59:14 dist-test-slave-glgz krb5kdc[26295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481554, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.67@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 03:59:14.349895 26379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:14.341481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:55935 (local address 127.15.185.126:34747)
0927 03:59:14.341544 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 03:59:14.341546 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:59:14.341760 (+ 214us) server_negotiation.cc:407] Connection header received
0927 03:59:14.342231 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:59:14.342232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:59:14.342242 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:59:14.342301 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:59:14.343676 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:14.344199 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:14.345197 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:14.345379 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:14.345429 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:14.347411 (+ 1982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:59:14.347412 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:59:14.347413 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:59:14.347419 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:59:14.348772 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:59:14.349150 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:59:14.349151 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:59:14.349151 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:59:14.349199 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:59:14.349392 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:59:14.349393 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:59:14.349393 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:59:14.349500 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:59:14.349550 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 03:59:14.349774 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 03:59:14.349832 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 03:59:14.350158 26801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:14.341463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34747 (local address 127.15.185.67:55935)
0927 03:59:14.341632 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 03:59:14.341674 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 03:59:14.342153 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:59:14.342400 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:59:14.342404 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:59:14.343282 (+ 878us) client_negotiation.cc:189] Negotiated authn=SASL
0927 03:59:14.343571 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:14.343576 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:14.344272 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:14.344273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:14.345126 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:14.345128 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:14.345446 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:14.345447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:14.345547 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:14.346011 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 03:59:14.346019 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 03:59:14.347335 (+ 1316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 03:59:14.348839 (+ 1504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:59:14.348845 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:59:14.348846 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:59:14.349081 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:59:14.349253 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 03:59:14.349254 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 03:59:14.349254 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 03:59:14.349345 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 03:59:14.349552 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 03:59:14.349556 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 03:59:14.349669 (+ 113us) client_negotiation.cc:769] Sending connection context
0927 03:59:14.349755 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 03:59:14.349869 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":94,"threads_started":1}
I20260927 03:59:14.350618 26799 heartbeater.cc:347] Connected to a master server at 127.15.185.126:34747
I20260927 03:59:14.350699 26799 heartbeater.cc:464] Registering TS with master...
I20260927 03:59:14.350884 26799 heartbeater.cc:511] Master 127.15.185.126:34747 requested a full tablet report, sending...
I20260927 03:59:14.351290 26340 ts_manager.cc:194] Registered new tserver with Master: 65bb143dae144cd581d2109cf741ba88 (127.15.185.67:43413)
I20260927 03:59:14.351657 26340 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.185.67@KRBTEST.COM'} at 127.15.185.67:55935
I20260927 03:59:14.354326 16101 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 03:59:14.361681 26379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:14.355712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32836 (local address 127.15.185.126:34747)
0927 03:59:14.355765 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 03:59:14.355767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:59:14.355842 (+ 75us) server_negotiation.cc:407] Connection header received
0927 03:59:14.355900 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:59:14.355901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:59:14.355909 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:59:14.355969 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:59:14.356504 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:14.356986 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:14.357721 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:14.357911 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:14.357996 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:14.359177 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:59:14.359178 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:59:14.359178 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:59:14.359185 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:59:14.360505 (+ 1320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:59:14.360894 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:59:14.360895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:59:14.360896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:59:14.360957 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:59:14.361153 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:59:14.361154 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:59:14.361154 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:59:14.361291 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:59:14.361395 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 03:59:14.361509 (+ 114us) server_negotiation.cc:299] Negotiation successful
0927 03:59:14.361583 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 03:59:14.363334 26340 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32836:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 03:59:14.364768 26340 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 03:59:14.371464 26807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:14.367011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.66:39567 (local address 127.0.0.1:32982)
0927 03:59:14.367192 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 03:59:14.367228 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 03:59:14.367287 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:59:14.367790 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:59:14.367791 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:59:14.367792 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:59:14.367841 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:14.367843 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:14.369026 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:14.369027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:14.370124 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:14.370126 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:14.371092 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:14.371093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:14.371193 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:14.371209 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 03:59:14.371260 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 03:59:14.371361 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":35,"thread_start_us":81,"threads_started":1}
I20260927 03:59:14.371773 26810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:14.367249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.65:42897 (local address 127.0.0.1:33204)
0927 03:59:14.367398 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 03:59:14.367405 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0927 03:59:14.367440 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:59:14.368030 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:59:14.368031 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:59:14.368033 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:59:14.368083 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:14.368084 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:14.369257 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:14.369258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:14.370450 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:14.370453 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:14.371523 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:14.371524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:14.371611 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:14.371615 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:59:14.371661 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 03:59:14.371707 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":8,"thread_start_us":34,"threads_started":1}
I20260927 03:59:14.371873 26809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:14.367154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32982 (local address 127.15.185.66:39567)
0927 03:59:14.367598 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0927 03:59:14.367602 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:59:14.367612 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:59:14.367644 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:59:14.367646 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:59:14.367659 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:59:14.367926 (+ 267us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:59:14.367998 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:14.368931 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:14.370232 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:14.370959 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:14.371673 (+ 714us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:14.371730 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 03:59:14.371755 (+ 25us) server_negotiation.cc:299] Negotiation successful
0927 03:59:14.371794 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":103,"threads_started":1}
I20260927 03:59:14.371862 26808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:14.367014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.67:43413 (local address 127.0.0.1:56320)
0927 03:59:14.367324 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0927 03:59:14.367333 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 03:59:14.367370 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:59:14.367737 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:59:14.367740 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:59:14.367743 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:59:14.367823 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:14.367825 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:14.369202 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:14.369203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:14.370450 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:14.370453 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:14.371640 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:14.371641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:14.371709 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:14.371713 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:59:14.371764 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 03:59:14.371801 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":66,"thread_start_us":79,"threads_started":1}
I20260927 03:59:14.372141 26811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:14.367367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33204 (local address 127.15.185.65:42897)
0927 03:59:14.367873 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0927 03:59:14.367877 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 03:59:14.367886 (+ 9us) server_negotiation.cc:407] Connection header received
0927 03:59:14.367921 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:59:14.367923 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:59:14.367944 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:59:14.368026 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:59:14.368156 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:14.369173 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:14.370564 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:14.371421 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:14.371610 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:14.371674 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 03:59:14.371700 (+ 26us) server_negotiation.cc:299] Negotiation successful
0927 03:59:14.371737 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":103,"threads_started":1}
I20260927 03:59:14.372539 26599 tablet_service.cc:1467] Processing CreateTablet for tablet d2e87e6136354dd389daf152631d93d7 (DEFAULT_TABLE table=test-table [id=ab7cf19c803d4d4e97c0e7797f029a22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:59:14.372735 26465 tablet_service.cc:1467] Processing CreateTablet for tablet d2e87e6136354dd389daf152631d93d7 (DEFAULT_TABLE table=test-table [id=ab7cf19c803d4d4e97c0e7797f029a22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:59:14.372881 26599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2e87e6136354dd389daf152631d93d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:59:14.373052 26465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2e87e6136354dd389daf152631d93d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:59:14.375136 26812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:14.367384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56320 (local address 127.15.185.67:43413)
0927 03:59:14.367565 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 03:59:14.367571 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 03:59:14.367581 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:59:14.367618 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:59:14.367620 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:59:14.367636 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:59:14.367718 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:59:14.367932 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:14.369106 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:14.370535 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:14.371443 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:14.371525 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:14.371603 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 03:59:14.371970 (+ 367us) server_negotiation.cc:299] Negotiation successful
0927 03:59:14.372021 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":87,"threads_started":1}
I20260927 03:59:14.375890 26813 tablet_bootstrap.cc:492] T d2e87e6136354dd389daf152631d93d7 P 52728bcd4f874803a2ecb114b3b19455: Bootstrap starting.
I20260927 03:59:14.375777 26733 tablet_service.cc:1467] Processing CreateTablet for tablet d2e87e6136354dd389daf152631d93d7 (DEFAULT_TABLE table=test-table [id=ab7cf19c803d4d4e97c0e7797f029a22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 03:59:14.376060 26814 tablet_bootstrap.cc:492] T d2e87e6136354dd389daf152631d93d7 P 1557db66f9a8494eaa4f6a8567ffb2a5: Bootstrap starting.
I20260927 03:59:14.376130 26733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2e87e6136354dd389daf152631d93d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:59:14.376463 26813 tablet_bootstrap.cc:654] T d2e87e6136354dd389daf152631d93d7 P 52728bcd4f874803a2ecb114b3b19455: Neither blocks nor log segments found. Creating new log.
I20260927 03:59:14.376674 26814 tablet_bootstrap.cc:654] T d2e87e6136354dd389daf152631d93d7 P 1557db66f9a8494eaa4f6a8567ffb2a5: Neither blocks nor log segments found. Creating new log.
I20260927 03:59:14.376921 26813 log.cc:824] T d2e87e6136354dd389daf152631d93d7 P 52728bcd4f874803a2ecb114b3b19455: Log is configured to *not* fsync() on all Append() calls
I20260927 03:59:14.377187 26814 log.cc:824] T d2e87e6136354dd389daf152631d93d7 P 1557db66f9a8494eaa4f6a8567ffb2a5: Log is configured to *not* fsync() on all Append() calls
I20260927 03:59:14.377599 26813 tablet_bootstrap.cc:492] T d2e87e6136354dd389daf152631d93d7 P 52728bcd4f874803a2ecb114b3b19455: No bootstrap required, opened a new log
I20260927 03:59:14.377660 26813 ts_tablet_manager.cc:1397] T d2e87e6136354dd389daf152631d93d7 P 52728bcd4f874803a2ecb114b3b19455: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:59:14.377920 26814 tablet_bootstrap.cc:492] T d2e87e6136354dd389daf152631d93d7 P 1557db66f9a8494eaa4f6a8567ffb2a5: No bootstrap required, opened a new log
I20260927 03:59:14.377974 26814 ts_tablet_manager.cc:1397] T d2e87e6136354dd389daf152631d93d7 P 1557db66f9a8494eaa4f6a8567ffb2a5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 03:59:14.378908 26813 raft_consensus.cc:348] T d2e87e6136354dd389daf152631d93d7 P 52728bcd4f874803a2ecb114b3b19455 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52728bcd4f874803a2ecb114b3b19455" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 39567 } } peers { permanent_uuid: "65bb143dae144cd581d2109cf741ba88" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 43413 } } peers { permanent_uuid: "1557db66f9a8494eaa4f6a8567ffb2a5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 42897 } }
I20260927 03:59:14.379022 26813 raft_consensus.cc:374] T d2e87e6136354dd389daf152631d93d7 P 52728bcd4f874803a2ecb114b3b19455 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:59:14.379091 26813 raft_consensus.cc:729] T d2e87e6136354dd389daf152631d93d7 P 52728bcd4f874803a2ecb114b3b19455 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52728bcd4f874803a2ecb114b3b19455, State: Initialized, Role: FOLLOWER
I20260927 03:59:14.379211 26813 consensus_queue.cc:260] T d2e87e6136354dd389daf152631d93d7 P 52728bcd4f874803a2ecb114b3b19455 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52728bcd4f874803a2ecb114b3b19455" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 39567 } } peers { permanent_uuid: "65bb143dae144cd581d2109cf741ba88" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 43413 } } peers { permanent_uuid: "1557db66f9a8494eaa4f6a8567ffb2a5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 42897 } }
I20260927 03:59:14.379460 26817 tablet_bootstrap.cc:492] T d2e87e6136354dd389daf152631d93d7 P 65bb143dae144cd581d2109cf741ba88: Bootstrap starting.
I20260927 03:59:14.379602 26813 ts_tablet_manager.cc:1428] T d2e87e6136354dd389daf152631d93d7 P 52728bcd4f874803a2ecb114b3b19455: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:59:14.379742 26665 heartbeater.cc:503] Master 127.15.185.126:34747 was elected leader, sending a full tablet report...
I20260927 03:59:14.380091 26817 tablet_bootstrap.cc:654] T d2e87e6136354dd389daf152631d93d7 P 65bb143dae144cd581d2109cf741ba88: Neither blocks nor log segments found. Creating new log.
I20260927 03:59:14.380494 26814 raft_consensus.cc:348] T d2e87e6136354dd389daf152631d93d7 P 1557db66f9a8494eaa4f6a8567ffb2a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52728bcd4f874803a2ecb114b3b19455" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 39567 } } peers { permanent_uuid: "65bb143dae144cd581d2109cf741ba88" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 43413 } } peers { permanent_uuid: "1557db66f9a8494eaa4f6a8567ffb2a5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 42897 } }
I20260927 03:59:14.380652 26814 raft_consensus.cc:374] T d2e87e6136354dd389daf152631d93d7 P 1557db66f9a8494eaa4f6a8567ffb2a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:59:14.380692 26814 raft_consensus.cc:729] T d2e87e6136354dd389daf152631d93d7 P 1557db66f9a8494eaa4f6a8567ffb2a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1557db66f9a8494eaa4f6a8567ffb2a5, State: Initialized, Role: FOLLOWER
I20260927 03:59:14.380707 26817 log.cc:824] T d2e87e6136354dd389daf152631d93d7 P 65bb143dae144cd581d2109cf741ba88: Log is configured to *not* fsync() on all Append() calls
I20260927 03:59:14.380843 26814 consensus_queue.cc:260] T d2e87e6136354dd389daf152631d93d7 P 1557db66f9a8494eaa4f6a8567ffb2a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52728bcd4f874803a2ecb114b3b19455" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 39567 } } peers { permanent_uuid: "65bb143dae144cd581d2109cf741ba88" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 43413 } } peers { permanent_uuid: "1557db66f9a8494eaa4f6a8567ffb2a5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 42897 } }
I20260927 03:59:14.381134 26814 ts_tablet_manager.cc:1428] T d2e87e6136354dd389daf152631d93d7 P 1557db66f9a8494eaa4f6a8567ffb2a5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 03:59:14.381147 26531 heartbeater.cc:503] Master 127.15.185.126:34747 was elected leader, sending a full tablet report...
I20260927 03:59:14.381554 26817 tablet_bootstrap.cc:492] T d2e87e6136354dd389daf152631d93d7 P 65bb143dae144cd581d2109cf741ba88: No bootstrap required, opened a new log
I20260927 03:59:14.381611 26817 ts_tablet_manager.cc:1397] T d2e87e6136354dd389daf152631d93d7 P 65bb143dae144cd581d2109cf741ba88: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:59:14.383198 26817 raft_consensus.cc:348] T d2e87e6136354dd389daf152631d93d7 P 65bb143dae144cd581d2109cf741ba88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52728bcd4f874803a2ecb114b3b19455" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 39567 } } peers { permanent_uuid: "65bb143dae144cd581d2109cf741ba88" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 43413 } } peers { permanent_uuid: "1557db66f9a8494eaa4f6a8567ffb2a5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 42897 } }
I20260927 03:59:14.383356 26817 raft_consensus.cc:374] T d2e87e6136354dd389daf152631d93d7 P 65bb143dae144cd581d2109cf741ba88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:59:14.383402 26817 raft_consensus.cc:729] T d2e87e6136354dd389daf152631d93d7 P 65bb143dae144cd581d2109cf741ba88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65bb143dae144cd581d2109cf741ba88, State: Initialized, Role: FOLLOWER
I20260927 03:59:14.383517 26817 consensus_queue.cc:260] T d2e87e6136354dd389daf152631d93d7 P 65bb143dae144cd581d2109cf741ba88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52728bcd4f874803a2ecb114b3b19455" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 39567 } } peers { permanent_uuid: "65bb143dae144cd581d2109cf741ba88" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 43413 } } peers { permanent_uuid: "1557db66f9a8494eaa4f6a8567ffb2a5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 42897 } }
I20260927 03:59:14.383793 26817 ts_tablet_manager.cc:1428] T d2e87e6136354dd389daf152631d93d7 P 65bb143dae144cd581d2109cf741ba88: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 03:59:14.383827 26799 heartbeater.cc:503] Master 127.15.185.126:34747 was elected leader, sending a full tablet report...
W20260927 03:59:14.434145 26666 tablet.cc:2369] T d2e87e6136354dd389daf152631d93d7 P 52728bcd4f874803a2ecb114b3b19455: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 03:59:14.537560 26532 tablet.cc:2369] T d2e87e6136354dd389daf152631d93d7 P 1557db66f9a8494eaa4f6a8567ffb2a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 03:59:14.591415 26800 tablet.cc:2369] T d2e87e6136354dd389daf152631d93d7 P 65bb143dae144cd581d2109cf741ba88: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 03:59:14.746929 26818 raft_consensus.cc:482] T d2e87e6136354dd389daf152631d93d7 P 52728bcd4f874803a2ecb114b3b19455 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 03:59:14.747081 26818 raft_consensus.cc:504] T d2e87e6136354dd389daf152631d93d7 P 52728bcd4f874803a2ecb114b3b19455 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52728bcd4f874803a2ecb114b3b19455" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 39567 } } peers { permanent_uuid: "65bb143dae144cd581d2109cf741ba88" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 43413 } } peers { permanent_uuid: "1557db66f9a8494eaa4f6a8567ffb2a5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 42897 } }
I20260927 03:59:14.747419 26818 leader_election.cc:288] T d2e87e6136354dd389daf152631d93d7 P 52728bcd4f874803a2ecb114b3b19455 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65bb143dae144cd581d2109cf741ba88 (127.15.185.67:43413), 1557db66f9a8494eaa4f6a8567ffb2a5 (127.15.185.65:42897)
I20260927 03:59:14.750344 26811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:14.747690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:53655 (local address 127.15.185.65:42897)
0927 03:59:14.747739 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 03:59:14.747741 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:59:14.747839 (+ 98us) server_negotiation.cc:407] Connection header received
0927 03:59:14.747919 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:59:14.747924 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:59:14.747931 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:59:14.747996 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:59:14.748173 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:14.748658 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:14.749653 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:14.749998 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:14.750052 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:14.750058 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0927 03:59:14.750226 (+ 168us) server_negotiation.cc:299] Negotiation successful
0927 03:59:14.750267 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 03:59:14.750370 26812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:14.747738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:50871 (local address 127.15.185.67:43413)
0927 03:59:14.747782 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 03:59:14.747784 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:59:14.747794 (+ 10us) server_negotiation.cc:407] Connection header received
0927 03:59:14.747915 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:59:14.747917 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:59:14.747926 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:59:14.748000 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 03:59:14.748201 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:14.748650 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:14.749655 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:14.750044 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:14.750114 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:14.750122 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 03:59:14.750266 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 03:59:14.750301 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 03:59:14.750385 26823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:14.747617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.65:42897 (local address 127.15.185.66:53655)
0927 03:59:14.747751 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 03:59:14.747762 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:59:14.747833 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:59:14.748011 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:59:14.748012 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:59:14.748014 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:59:14.748106 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:14.748107 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:14.748764 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:14.748764 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:14.749561 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:14.749563 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:14.750079 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:14.750081 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:14.750151 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:14.750155 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:59:14.750211 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 03:59:14.750271 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":10,"thread_start_us":56,"threads_started":1}
I20260927 03:59:14.750391 26822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:14.747614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.67:43413 (local address 127.15.185.66:50871)
0927 03:59:14.747740 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 03:59:14.747752 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 03:59:14.747832 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:59:14.748005 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 03:59:14.748006 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 03:59:14.748008 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 03:59:14.748111 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:14.748113 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:14.748728 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:14.748729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:14.749560 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 03:59:14.749563 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 03:59:14.750124 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 03:59:14.750125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 03:59:14.750176 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:14.750180 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 03:59:14.750249 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 03:59:14.750286 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":18,"thread_start_us":66,"threads_started":1}
I20260927 03:59:14.750777 26485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2e87e6136354dd389daf152631d93d7" candidate_uuid: "52728bcd4f874803a2ecb114b3b19455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1557db66f9a8494eaa4f6a8567ffb2a5" is_pre_election: true
I20260927 03:59:14.750927 26485 raft_consensus.cc:2454] T d2e87e6136354dd389daf152631d93d7 P 1557db66f9a8494eaa4f6a8567ffb2a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52728bcd4f874803a2ecb114b3b19455 in term 0.
I20260927 03:59:14.750906 26753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2e87e6136354dd389daf152631d93d7" candidate_uuid: "52728bcd4f874803a2ecb114b3b19455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65bb143dae144cd581d2109cf741ba88" is_pre_election: true
I20260927 03:59:14.751097 26753 raft_consensus.cc:2454] T d2e87e6136354dd389daf152631d93d7 P 65bb143dae144cd581d2109cf741ba88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52728bcd4f874803a2ecb114b3b19455 in term 0.
I20260927 03:59:14.751158 26552 leader_election.cc:302] T d2e87e6136354dd389daf152631d93d7 P 52728bcd4f874803a2ecb114b3b19455 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1557db66f9a8494eaa4f6a8567ffb2a5, 52728bcd4f874803a2ecb114b3b19455; no voters:
I20260927 03:59:14.751286 26818 raft_consensus.cc:2781] T d2e87e6136354dd389daf152631d93d7 P 52728bcd4f874803a2ecb114b3b19455 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 03:59:14.751350 26818 raft_consensus.cc:482] T d2e87e6136354dd389daf152631d93d7 P 52728bcd4f874803a2ecb114b3b19455 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 03:59:14.751380 26818 raft_consensus.cc:3037] T d2e87e6136354dd389daf152631d93d7 P 52728bcd4f874803a2ecb114b3b19455 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:59:14.752080 26818 raft_consensus.cc:504] T d2e87e6136354dd389daf152631d93d7 P 52728bcd4f874803a2ecb114b3b19455 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52728bcd4f874803a2ecb114b3b19455" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 39567 } } peers { permanent_uuid: "65bb143dae144cd581d2109cf741ba88" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 43413 } } peers { permanent_uuid: "1557db66f9a8494eaa4f6a8567ffb2a5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 42897 } }
I20260927 03:59:14.752215 26818 leader_election.cc:288] T d2e87e6136354dd389daf152631d93d7 P 52728bcd4f874803a2ecb114b3b19455 [CANDIDATE]: Term 1 election: Requested vote from peers 65bb143dae144cd581d2109cf741ba88 (127.15.185.67:43413), 1557db66f9a8494eaa4f6a8567ffb2a5 (127.15.185.65:42897)
I20260927 03:59:14.752357 26485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2e87e6136354dd389daf152631d93d7" candidate_uuid: "52728bcd4f874803a2ecb114b3b19455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1557db66f9a8494eaa4f6a8567ffb2a5"
I20260927 03:59:14.752369 26753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2e87e6136354dd389daf152631d93d7" candidate_uuid: "52728bcd4f874803a2ecb114b3b19455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65bb143dae144cd581d2109cf741ba88"
I20260927 03:59:14.752431 26753 raft_consensus.cc:3037] T d2e87e6136354dd389daf152631d93d7 P 65bb143dae144cd581d2109cf741ba88 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:59:14.752431 26485 raft_consensus.cc:3037] T d2e87e6136354dd389daf152631d93d7 P 1557db66f9a8494eaa4f6a8567ffb2a5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:59:14.753289 26485 raft_consensus.cc:2454] T d2e87e6136354dd389daf152631d93d7 P 1557db66f9a8494eaa4f6a8567ffb2a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52728bcd4f874803a2ecb114b3b19455 in term 1.
I20260927 03:59:14.753289 26753 raft_consensus.cc:2454] T d2e87e6136354dd389daf152631d93d7 P 65bb143dae144cd581d2109cf741ba88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52728bcd4f874803a2ecb114b3b19455 in term 1.
I20260927 03:59:14.753443 26552 leader_election.cc:302] T d2e87e6136354dd389daf152631d93d7 P 52728bcd4f874803a2ecb114b3b19455 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1557db66f9a8494eaa4f6a8567ffb2a5, 52728bcd4f874803a2ecb114b3b19455; no voters:
I20260927 03:59:14.753538 26818 raft_consensus.cc:2781] T d2e87e6136354dd389daf152631d93d7 P 52728bcd4f874803a2ecb114b3b19455 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:59:14.753667 26818 raft_consensus.cc:686] T d2e87e6136354dd389daf152631d93d7 P 52728bcd4f874803a2ecb114b3b19455 [term 1 LEADER]: Becoming Leader. State: Replica: 52728bcd4f874803a2ecb114b3b19455, State: Running, Role: LEADER
I20260927 03:59:14.753753 26818 consensus_queue.cc:237] T d2e87e6136354dd389daf152631d93d7 P 52728bcd4f874803a2ecb114b3b19455 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52728bcd4f874803a2ecb114b3b19455" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 39567 } } peers { permanent_uuid: "65bb143dae144cd581d2109cf741ba88" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 43413 } } peers { permanent_uuid: "1557db66f9a8494eaa4f6a8567ffb2a5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 42897 } }
I20260927 03:59:14.754554 26340 catalog_manager.cc:5495] T d2e87e6136354dd389daf152631d93d7 P 52728bcd4f874803a2ecb114b3b19455 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52728bcd4f874803a2ecb114b3b19455 (127.15.185.66). New cstate: current_term: 1 leader_uuid: "52728bcd4f874803a2ecb114b3b19455" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52728bcd4f874803a2ecb114b3b19455" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 39567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65bb143dae144cd581d2109cf741ba88" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 43413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1557db66f9a8494eaa4f6a8567ffb2a5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 42897 } health_report { overall_health: UNKNOWN } } }
I20260927 03:59:14.815514 26809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:14.813029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32994 (local address 127.15.185.66:39567)
0927 03:59:14.813085 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 03:59:14.813087 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 03:59:14.813099 (+ 12us) server_negotiation.cc:407] Connection header received
0927 03:59:14.813133 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:59:14.813134 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:59:14.813151 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:59:14.813223 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 03:59:14.813376 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:14.813861 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:14.814633 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:14.814822 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:14.814905 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:14.815112 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 03:59:14.815244 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 03:59:14.815345 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 03:59:14.815394 (+ 49us) server_negotiation.cc:299] Negotiation successful
0927 03:59:14.815414 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 03:59:14.817350 26485 raft_consensus.cc:1261] T d2e87e6136354dd389daf152631d93d7 P 1557db66f9a8494eaa4f6a8567ffb2a5 [term 1 FOLLOWER]: Refusing update from remote peer 52728bcd4f874803a2ecb114b3b19455: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:59:14.817389 26753 raft_consensus.cc:1261] T d2e87e6136354dd389daf152631d93d7 P 65bb143dae144cd581d2109cf741ba88 [term 1 FOLLOWER]: Refusing update from remote peer 52728bcd4f874803a2ecb114b3b19455: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 03:59:14.817687 26824 consensus_queue.cc:1035] T d2e87e6136354dd389daf152631d93d7 P 52728bcd4f874803a2ecb114b3b19455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1557db66f9a8494eaa4f6a8567ffb2a5" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:59:14.817842 26818 consensus_queue.cc:1035] T d2e87e6136354dd389daf152631d93d7 P 52728bcd4f874803a2ecb114b3b19455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65bb143dae144cd581d2109cf741ba88" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 43413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 03:59:14.819953 26827 mvcc.cc:205] Tried to move back new op lower bound from 7333812448529391616 to 7333812448272044032. Current Snapshot: MvccSnapshot[applied={T|T < 7333812448529391616}]
I20260927 03:59:14.820222 26828 mvcc.cc:205] Tried to move back new op lower bound from 7333812448529391616 to 7333812448272044032. Current Snapshot: MvccSnapshot[applied={T|T < 7333812448529391616}]
I20260927 03:59:14.821262 26829 mvcc.cc:205] Tried to move back new op lower bound from 7333812448529391616 to 7333812448272044032. Current Snapshot: MvccSnapshot[applied={T|T < 7333812448529391616 or (T in {7333812448529391616})}]
I20260927 03:59:14.822271 26339 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:59:14.822369 26339 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 03:59:14.823096 26339 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cce83b5c7e1f44e3ba08932330d49b8c: Sending DeleteTablet for 3 replicas of tablet d2e87e6136354dd389daf152631d93d7
I20260927 03:59:14.823381 26599 tablet_service.cc:1514] Processing DeleteTablet for tablet d2e87e6136354dd389daf152631d93d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:59:14 UTC) from {username='kudu', principal='kudu/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:32982
I20260927 03:59:14.823469 26465 tablet_service.cc:1514] Processing DeleteTablet for tablet d2e87e6136354dd389daf152631d93d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:59:14 UTC) from {username='kudu', principal='kudu/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:33204
I20260927 03:59:14.823690 26838 tablet_replica.cc:330] T d2e87e6136354dd389daf152631d93d7 P 1557db66f9a8494eaa4f6a8567ffb2a5: stopping tablet replica
I20260927 03:59:14.823685 26733 tablet_service.cc:1514] Processing DeleteTablet for tablet d2e87e6136354dd389daf152631d93d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 03:59:14 UTC) from {username='kudu', principal='kudu/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:56320
I20260927 03:59:14.823843 26838 raft_consensus.cc:2229] T d2e87e6136354dd389daf152631d93d7 P 1557db66f9a8494eaa4f6a8567ffb2a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:59:14.823873 26839 tablet_replica.cc:330] T d2e87e6136354dd389daf152631d93d7 P 65bb143dae144cd581d2109cf741ba88: stopping tablet replica
I20260927 03:59:14.823975 26839 raft_consensus.cc:2229] T d2e87e6136354dd389daf152631d93d7 P 65bb143dae144cd581d2109cf741ba88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 03:59:14.823987 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 26402
I20260927 03:59:14.824090 26839 raft_consensus.cc:2258] T d2e87e6136354dd389daf152631d93d7 P 65bb143dae144cd581d2109cf741ba88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 03:59:14.825146 26839 ts_tablet_manager.cc:1905] T d2e87e6136354dd389daf152631d93d7 P 65bb143dae144cd581d2109cf741ba88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:59:14.827126 26839 ts_tablet_manager.cc:1918] T d2e87e6136354dd389daf152631d93d7 P 65bb143dae144cd581d2109cf741ba88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 03:59:14.827199 26839 log.cc:1196] T d2e87e6136354dd389daf152631d93d7 P 65bb143dae144cd581d2109cf741ba88: Deleting WAL directory at /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481498018407-16101-0/minicluster-data/ts-2/wal/wals/d2e87e6136354dd389daf152631d93d7
I20260927 03:59:14.827455 26839 ts_tablet_manager.cc:1939] T d2e87e6136354dd389daf152631d93d7 P 65bb143dae144cd581d2109cf741ba88: Deleting consensus metadata
I20260927 03:59:14.827865 26327 catalog_manager.cc:4841] TS 65bb143dae144cd581d2109cf741ba88 (127.15.185.67:43413): tablet d2e87e6136354dd389daf152631d93d7 (table test-table [id=ab7cf19c803d4d4e97c0e7797f029a22]) successfully deleted
W20260927 03:59:14.831216 26324 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.185.65:42897 (error 108)
I20260927 03:59:14.831354 26837 tablet_replica.cc:330] T d2e87e6136354dd389daf152631d93d7 P 52728bcd4f874803a2ecb114b3b19455: stopping tablet replica
I20260927 03:59:14.831514 26837 raft_consensus.cc:2229] T d2e87e6136354dd389daf152631d93d7 P 52728bcd4f874803a2ecb114b3b19455 [term 1 LEADER]: Raft consensus shutting down.
I20260927 03:59:14.831650 26837 raft_consensus.cc:2258] T d2e87e6136354dd389daf152631d93d7 P 52728bcd4f874803a2ecb114b3b19455 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 03:59:14.831843 26808 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 03:59:14.831596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.65:42897 (local address 127.0.0.1:33220)
0927 03:59:14.831644 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 03:59:14.831655 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 03:59:14.831691 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 03:59:14.831798 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.65:42897: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 03:59:14.831974 26324 catalog_manager.cc:4584] TS 1557db66f9a8494eaa4f6a8567ffb2a5 (127.15.185.65:42897): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d2e87e6136354dd389daf152631d93d7: Network error: Client connection negotiation failed: client connection to 127.15.185.65:42897: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 03:59:14.832208 26837 ts_tablet_manager.cc:1905] T d2e87e6136354dd389daf152631d93d7 P 52728bcd4f874803a2ecb114b3b19455: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 03:59:14.832373 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 26536
I20260927 03:59:14.838101 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 26670
I20260927 03:59:14.838189 26808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:14.838044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.66:39567 (local address 127.0.0.1:32998)
0927 03:59:14.838094 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 03:59:14.838149 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.66:39567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260927 03:59:14.842231 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 26308
2026-09-27T03:59:14Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2296 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 03:59:14 dist-test-slave-glgz krb5kdc[26843](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 03:59:14 dist-test-slave-glgz krb5kdc[26843](info): set up 2 sockets
Sep 27 03:59:14 dist-test-slave-glgz krb5kdc[26843](info): commencing operation
krb5kdc: starting...
W20260927 03:59:15.877094 16101 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Sep 27 03:59:15 dist-test-slave-glgz krb5kdc[26843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T03:59:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T03:59:15Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.15.185.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.15.185.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.15.185.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/krb5kdc/rangeradmin_127.15.185.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.15.185.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/krb5kdc/rangeradmin_127.15.185.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.15.185.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.15.185.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.15.185.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/krb5kdc/rangerlookup_127.15.185.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.15.185.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/krb5kdc/rangerlookup_127.15.185.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.15.185.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.15.185.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.15.185.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/krb5kdc/HTTP_127.15.185.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.15.185.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/krb5kdc/HTTP_127.15.185.84@KRBTEST.COM.keytab.
I20260927 03:59:16.046149 16101 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskTmmW3q/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/postgres -l logfile start
2026-09-27 03:59:18.878 UTC [26873] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 03:59:18.878 UTC [26873] LOG: listening on IPv4 address "127.15.185.84", port 59029
2026-09-27 03:59:18.882 UTC [26873] LOG: listening on Unix socket "/tmp/.s.PGSQL.59029"
2026-09-27 03:59:18.917 UTC [26876] LOG: database system was shut down at 2026-09-27 03:59:17 UTC
2026-09-27 03:59:18.921 UTC [26873] LOG: database system is ready to accept connections
I20260927 03:59:19.948800 16101 mini_postgres.cc:96] Postgres bound to 59029
2026-09-27 03:59:19.955 UTC [26884] FATAL: database "slave" does not exist
127.15.185.84:59029 - accepting connections
I20260927 03:59:19.955636 16101 mini_ranger.cc:161] Starting Ranger...
I20260927 03:59:19.970767 16101 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 03:59:20.544936 16101 mini_ranger.cc:87] Created ranger Postgres database
I20260927 03:59:20.545047 16101 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ranger-admin
2026-09-27 03:59:20,834 [I] DB FLAVOR :POSTGRES
2026-09-27 03:59:20,835 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 03:59:20,835 [I] Checking connection..
2026-09-27 03:59:20,835 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTmmW3q/build/release/bin/postgresql.jar:/tmp/dist-test-taskTmmW3q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.185.84:59029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 03:59:21,098 [I] Checking connection passed.
2026-09-27 03:59:21,099 [I] --------- Verifying version history table ---------
2026-09-27 03:59:21,099 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTmmW3q/build/release/bin/postgresql.jar:/tmp/dist-test-taskTmmW3q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.185.84:59029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 03:59:21,362 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 03:59:21,363 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTmmW3q/build/release/bin/postgresql.jar:/tmp/dist-test-taskTmmW3q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.185.84:59029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 03:59:21,627 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 03:59:21,627 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 03:59:21,627 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTmmW3q/build/release/bin/postgresql.jar:/tmp/dist-test-taskTmmW3q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.185.84:59029/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskTmmW3q/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 03:59:21.865 UTC [26948] WARNING: there is no transaction in progress
2026-09-27 03:59:21,871 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 03:59:21,872 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTmmW3q/build/release/bin/postgresql.jar:/tmp/dist-test-taskTmmW3q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.185.84:59029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 03:59:22,106 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 03:59:22,106 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 03:59:22,107 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTmmW3q/build/release/bin/postgresql.jar:/tmp/dist-test-taskTmmW3q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.185.84:59029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 03:59:22,347 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTmmW3q/build/release/bin/postgresql.jar:/tmp/dist-test-taskTmmW3q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.185.84:59029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 03:59:22,585 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTmmW3q/build/release/bin/postgresql.jar:/tmp/dist-test-taskTmmW3q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.185.84:59029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-glgz.c.gcp-upstream.internal','N') ;"
2026-09-27 03:59:22,818 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTmmW3q/build/release/bin/postgresql.jar:/tmp/dist-test-taskTmmW3q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.185.84:59029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 03:59:23,081 [I] Table x_portal_user does not exist in database ranger
2026-09-27 03:59:23,081 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTmmW3q/build/release/bin/postgresql.jar:/tmp/dist-test-taskTmmW3q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.185.84:59029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 03:59:23,330 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 03:59:23,330 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTmmW3q/build/release/bin/postgresql.jar:/tmp/dist-test-taskTmmW3q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.185.84:59029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 03:59:23,592 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTmmW3q/build/release/bin/postgresql.jar:/tmp/dist-test-taskTmmW3q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.185.84:59029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 03:59:23,851 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 03:59:23,852 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTmmW3q/build/release/bin/postgresql.jar:/tmp/dist-test-taskTmmW3q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.185.84:59029/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskTmmW3q/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 03:59:24.827 UTC [27056] WARNING: there is no transaction in progress
2026-09-27 03:59:24.844 UTC [27056] WARNING: there is no transaction in progress
2026-09-27 03:59:24.860 UTC [27056] WARNING: there is no transaction in progress
2026-09-27 03:59:24.876 UTC [27056] WARNING: there is no transaction in progress
2026-09-27 03:59:24.893 UTC [27056] WARNING: there is no transaction in progress
2026-09-27 03:59:24.912 UTC [27056] WARNING: there is no transaction in progress
2026-09-27 03:59:24.970 UTC [27056] WARNING: there is no transaction in progress
2026-09-27 03:59:24.977 UTC [27056] WARNING: there is no transaction in progress
2026-09-27 03:59:24.992 UTC [27056] WARNING: there is no transaction in progress
2026-09-27 03:59:25.003 UTC [27056] WARNING: there is no transaction in progress
2026-09-27 03:59:25.014 UTC [27056] WARNING: there is no transaction in progress
2026-09-27 03:59:25.029 UTC [27056] WARNING: there is no transaction in progress
2026-09-27 03:59:25.038 UTC [27056] WARNING: there is no transaction in progress
2026-09-27 03:59:25.045 UTC [27056] WARNING: there is no transaction in progress
2026-09-27 03:59:25.052 UTC [27056] WARNING: there is no transaction in progress
2026-09-27 03:59:25.059 UTC [27056] WARNING: there is no transaction in progress
2026-09-27 03:59:25,707 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 03:59:25,708 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTmmW3q/build/release/bin/postgresql.jar:/tmp/dist-test-taskTmmW3q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.185.84:59029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 03:59:25,948 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 03:59:25,948 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTmmW3q/build/release/bin/postgresql.jar:/tmp/dist-test-taskTmmW3q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.185.84:59029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 03:59:26,209 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 03:59:26,210 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTmmW3q/build/release/bin/postgresql.jar:/tmp/dist-test-taskTmmW3q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.185.84:59029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 03:59:26,461 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 03:59:26,461 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTmmW3q/build/release/bin/postgresql.jar:/tmp/dist-test-taskTmmW3q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.185.84:59029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 03:59:26,721 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTmmW3q/build/release/bin/postgresql.jar:/tmp/dist-test-taskTmmW3q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.185.84:59029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 03:59:26,971 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTmmW3q/build/release/bin/postgresql.jar:/tmp/dist-test-taskTmmW3q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.185.84:59029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-glgz.c.gcp-upstream.internal';"
2026-09-27 03:59:27,230 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 03:59:27,230 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTmmW3q/build/release/bin/postgresql.jar:/tmp/dist-test-taskTmmW3q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.185.84:59029/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 03:59:27,477 [I] DB_PATCHES have already been applied
I20260927 03:59:27.482751 16101 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ranger-admin:/tmp/dist-test-taskTmmW3q/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskTmmW3q/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskTmmW3q/build/release/bin/postgresql.jar:/tmp/dist-test-taskTmmW3q/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 03:59:27.482827 16101 mini_ranger.cc:193] Using host: 127.15.185.84
I20260927 03:59:27.485922 16101 mini_ranger.cc:227] Ranger admin URL: http://127.15.185.84:58185
Sep 27, 2026 3:59:27 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 3:59:27 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 3:59:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 3:59:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 3:59:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 3:59:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 3:59:28 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 03:59:28.583329 16101 mini_ranger.cc:160] Time spent starting Ranger: real 8.628s user 0.000s sys 0.009s
Sep 27, 2026 3:59:49 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 3:59:49 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 3:59:49 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 3:59:49 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 3:59:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 3:59:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 3:59:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 3:59:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 3:59:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 3:59:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 3:59:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 3:59:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 3:59:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 3:59:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 3:59:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 3:59:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 3:59:49 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 3:59:50 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 03:59:50.761118 16101 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.15.185.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.15.185.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.15.185.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/krb5kdc/rangerkms_127.15.185.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.15.185.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/krb5kdc/rangerkms_127.15.185.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.15.185.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.15.185.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.15.185.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/krb5kdc/HTTP_127.15.185.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.15.185.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/krb5kdc/HTTP_127.15.185.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 03:59:50 dist-test-slave-glgz krb5kdc[26843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481590, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 03:59:50.851928 16101 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 03:59:50.861672 16101 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 03:59:51.820626 16101 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 03:59:51.820734 16101 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ranger-kms
I20260927 03:59:51.820755 16101 mini_ranger_kms.cc:227] Using postgres at 127.15.185.84:59029
2026-09-27 03:59:52,013 [I] DB FLAVOR :POSTGRES
2026-09-27 03:59:52,014 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 03:59:52,014 [I] Checking connection
2026-09-27 03:59:52,014 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTmmW3q/build/release/bin/postgresql.jar:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.185.84:59029/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 03:59:52,255 [I] connection success
2026-09-27 03:59:52,255 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 03:59:52,255 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 03:59:52,255 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTmmW3q/build/release/bin/postgresql.jar:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.185.84:59029/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 03:59:52,489 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 03:59:52,489 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 03:59:52,489 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 03:59:52,490 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTmmW3q/build/release/bin/postgresql.jar:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.185.84:59029/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 03:59:52,773 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 03:59:53.281684 16101 mini_ranger_kms.cc:310] Created kms service
I20260927 03:59:53.495282 16101 mini_ranger_kms.cc:326] Created kudu user
I20260927 03:59:53.539443 16101 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 03:59:53 dist-test-slave-glgz krb5kdc[26843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481590, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.15.185.84@KRBTEST.COM
I20260927 03:59:53.924510 16101 mini_ranger_kms.cc:384] Added ranger policy
I20260927 03:59:53.924626 16101 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ranger-kms:/tmp/dist-test-taskTmmW3q/build/release/bin/postgresql.jar:/tmp/dist-test-taskTmmW3q/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskTmmW3q/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 03:59:53.924649 16101 mini_ranger_kms.cc:240] Using host: 127.15.185.84
I20260927 03:59:53.928417 16101 mini_ranger_kms.cc:276] Ranger KMS PID: 27236
I20260927 03:59:53.928556 16101 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.15.185.84:59725
Sep 27, 2026 3:59:55 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 3:59:55 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 3:59:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 3:59:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 3:59:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 3:59:55 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 27 03:59:56 dist-test-slave-glgz krb5kdc[26843](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790481596, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.185.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:59:56.182323 16101 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.330s user 0.001s sys 0.174s
I20260927 03:59:56.182435 16101 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 03:59:56.182524 16101 mini_ranger_kms.cc:401] 127.15.185.84:59725/kms/v1/keys
Sep 27 03:59:56 dist-test-slave-glgz krb5kdc[26843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481596, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.185.84@KRBTEST.COM for HTTP/127.15.185.84@KRBTEST.COM
Sep 27 03:59:56 dist-test-slave-glgz krb5kdc[26843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481596, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.185.84@KRBTEST.COM for HTTP/127.15.185.84@KRBTEST.COM
Sep 27 03:59:59 dist-test-slave-glgz krb5kdc[26843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.15.185.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab.
Entry for principal kudu/127.15.185.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab.
I20260927 03:59:59.130201 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:36961
--webserver_interface=127.15.185.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:35299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.185.126:36961
--ranger_config_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.185.84:59725/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 03:59:59.214918 27287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 03:59:59.215193 27287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 03:59:59.215219 27287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 03:59:59.216414 27287 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 03:59:59.216455 27287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 03:59:59.216470 27287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 03:59:59.216483 27287 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 03:59:59.216495 27287 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 03:59:59.218016 27287 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:35299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/master-0/wal
--ranger_kms_url=127.15.185.84:59725/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.15.185.126:36961
--ranger_config_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:36961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27287
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 03:59:59.218207 27287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 03:59:59.218386 27287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 03:59:59.224093 27292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:59:59.224074 27293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 03:59:59.224069 27295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 03:59:59.224216 27287 server_base.cc:1034] running on GCE node
I20260927 03:59:59.224638 27287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 03:59:59.225014 27287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 03:59:59.226188 27287 hybrid_clock.cc:648] HybridClock initialized: now 1790481599226166 us; error 37 us; skew 500 ppm
Sep 27 03:59:59 dist-test-slave-glgz krb5kdc[26843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481599, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 03:59:59.233150 27287 init.cc:377] Logged in from keytab as kudu/127.15.185.126@KRBTEST.COM (short username kudu)
I20260927 03:59:59.233696 27287 webserver.cc:460] Webserver started at http://127.15.185.126:41341/ using document root <none> and password file <none>
I20260927 03:59:59.233955 27287 fs_manager.cc:359] Metadata directory not provided
I20260927 03:59:59.234019 27287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 03:59:59.234182 27287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 03:59:59 dist-test-slave-glgz krb5kdc[26843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481599, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.126@KRBTEST.COM for HTTP/127.15.185.84@KRBTEST.COM
I20260927 03:59:59.300143 27287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/master-0/data/instance:
uuid: "27c553f8b4a34b479edb3fd8491e63b1"
format_stamp: "Formatted at 2026-09-27 03:59:59 on dist-test-slave-glgz"
server_key: "ff27a9c2305685b18d6396a1103e2047"
server_key_iv: "0ac4c1f01650ae1a27e4b30c17a22d46"
server_key_version: "kuduclusterkey@0"
I20260927 03:59:59.300678 27287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/master-0/wal/instance:
uuid: "27c553f8b4a34b479edb3fd8491e63b1"
format_stamp: "Formatted at 2026-09-27 03:59:59 on dist-test-slave-glgz"
server_key: "ff27a9c2305685b18d6396a1103e2047"
server_key_iv: "0ac4c1f01650ae1a27e4b30c17a22d46"
server_key_version: "kuduclusterkey@0"
I20260927 03:59:59.302727 27287 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260927 03:59:59.318239 27302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 03:59:59.318511 27287 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 03:59:59.318650 27287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/master-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/master-0/wal
uuid: "27c553f8b4a34b479edb3fd8491e63b1"
format_stamp: "Formatted at 2026-09-27 03:59:59 on dist-test-slave-glgz"
server_key: "ff27a9c2305685b18d6396a1103e2047"
server_key_iv: "0ac4c1f01650ae1a27e4b30c17a22d46"
server_key_version: "kuduclusterkey@0"
I20260927 03:59:59.318781 27287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 03:59:59.327345 27287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 03:59:59.329406 27287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 03:59:59.329573 27287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 03:59:59.335127 27287 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.126:36961
I20260927 03:59:59.335235 27354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.126:36961 every 8 connection(s)
I20260927 03:59:59.335574 27287 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
I20260927 03:59:59.336270 27355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 03:59:59.336501 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 27287
I20260927 03:59:59.336617 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/master-0/wal/instance
I20260927 03:59:59.336833 16101 external_mini_cluster.cc:1428] Setting key d50d83e81a7caf9ba749bc8b3a140a6d
I20260927 03:59:59.339879 27355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27c553f8b4a34b479edb3fd8491e63b1: Bootstrap starting.
I20260927 03:59:59.340659 27355 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27c553f8b4a34b479edb3fd8491e63b1: Neither blocks nor log segments found. Creating new log.
I20260927 03:59:59.341001 27355 log.cc:824] T 00000000000000000000000000000000 P 27c553f8b4a34b479edb3fd8491e63b1: Log is configured to *not* fsync() on all Append() calls
I20260927 03:59:59.341820 27355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27c553f8b4a34b479edb3fd8491e63b1: No bootstrap required, opened a new log
I20260927 03:59:59.344802 27355 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27c553f8b4a34b479edb3fd8491e63b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27c553f8b4a34b479edb3fd8491e63b1" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 36961 } }
I20260927 03:59:59.345005 27355 raft_consensus.cc:374] T 00000000000000000000000000000000 P 27c553f8b4a34b479edb3fd8491e63b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 03:59:59.345047 27355 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27c553f8b4a34b479edb3fd8491e63b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27c553f8b4a34b479edb3fd8491e63b1, State: Initialized, Role: FOLLOWER
I20260927 03:59:59.345248 27355 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27c553f8b4a34b479edb3fd8491e63b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27c553f8b4a34b479edb3fd8491e63b1" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 36961 } }
I20260927 03:59:59.345395 27355 raft_consensus.cc:388] T 00000000000000000000000000000000 P 27c553f8b4a34b479edb3fd8491e63b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 03:59:59.345445 27355 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27c553f8b4a34b479edb3fd8491e63b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 03:59:59.345492 27355 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27c553f8b4a34b479edb3fd8491e63b1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 03:59:59.346280 27355 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27c553f8b4a34b479edb3fd8491e63b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27c553f8b4a34b479edb3fd8491e63b1" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 36961 } }
I20260927 03:59:59.346406 27355 leader_election.cc:302] T 00000000000000000000000000000000 P 27c553f8b4a34b479edb3fd8491e63b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 27c553f8b4a34b479edb3fd8491e63b1; no voters:
I20260927 03:59:59.346621 27355 leader_election.cc:288] T 00000000000000000000000000000000 P 27c553f8b4a34b479edb3fd8491e63b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 03:59:59.346648 27360 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 27c553f8b4a34b479edb3fd8491e63b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 03:59:59.346892 27360 raft_consensus.cc:686] T 00000000000000000000000000000000 P 27c553f8b4a34b479edb3fd8491e63b1 [term 1 LEADER]: Becoming Leader. State: Replica: 27c553f8b4a34b479edb3fd8491e63b1, State: Running, Role: LEADER
Sep 27 03:59:59 dist-test-slave-glgz krb5kdc[26843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 03:59:59.347028 27355 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27c553f8b4a34b479edb3fd8491e63b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 03:59:59.347556 27360 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27c553f8b4a34b479edb3fd8491e63b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27c553f8b4a34b479edb3fd8491e63b1" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 36961 } }
I20260927 03:59:59.347924 27361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27c553f8b4a34b479edb3fd8491e63b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "27c553f8b4a34b479edb3fd8491e63b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27c553f8b4a34b479edb3fd8491e63b1" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 36961 } } }
I20260927 03:59:59.348007 27361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27c553f8b4a34b479edb3fd8491e63b1 [sys.catalog]: This master's current role is: LEADER
I20260927 03:59:59.348429 27362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27c553f8b4a34b479edb3fd8491e63b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27c553f8b4a34b479edb3fd8491e63b1. Latest consensus state: current_term: 1 leader_uuid: "27c553f8b4a34b479edb3fd8491e63b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27c553f8b4a34b479edb3fd8491e63b1" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 36961 } } }
I20260927 03:59:59.348502 27362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27c553f8b4a34b479edb3fd8491e63b1 [sys.catalog]: This master's current role is: LEADER
I20260927 03:59:59.348636 27368 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 03:59:59.349093 27368 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 03:59:59.349390 27355 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 03:59:59.354580 27358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 03:59:59.337560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54414 (local address 127.15.185.126:36961)
0927 03:59:59.339182 (+ 1622us) server_negotiation.cc:206] Beginning negotiation
0927 03:59:59.339193 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 03:59:59.339227 (+ 34us) server_negotiation.cc:407] Connection header received
0927 03:59:59.339765 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 03:59:59.339774 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 03:59:59.339860 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 03:59:59.340870 (+ 1010us) server_negotiation.cc:226] Negotiated authn=SASL
0927 03:59:59.340988 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:59.342334 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:59.343432 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 03:59:59.344453 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 03:59:59.344573 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 03:59:59.347477 (+ 2904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 03:59:59.347482 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 03:59:59.347482 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 03:59:59.347497 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 03:59:59.349618 (+ 2121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:59:59.351288 (+ 1670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:59:59.351345 (+ 57us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:59:59.351345 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:59:59.351777 (+ 432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 03:59:59.352063 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 03:59:59.352063 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 03:59:59.352064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 03:59:59.352785 (+ 721us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 03:59:59.353544 (+ 759us) server_negotiation.cc:1035] Waiting for connection context
0927 03:59:59.353650 (+ 106us) server_negotiation.cc:299] Negotiation successful
0927 03:59:59.354225 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1568,"thread_start_us":91,"threads_started":1}
I20260927 03:59:59.357086 27378 mvcc.cc:205] Tried to move back new op lower bound from 7333812630961676288 to 7333812630927937536. Current Snapshot: MvccSnapshot[applied={T|T < 7333812630961676288}]
I20260927 03:59:59.357522 27368 catalog_manager.cc:1347] Generated new cluster ID: db116de3568e4ae1a27494ae2007cf24
I20260927 03:59:59.357573 27368 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 03:59:59.373737 27368 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 03:59:59.374553 27368 catalog_manager.cc:1504] Loading token signing keys...
I20260927 03:59:59.382752 27368 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 27c553f8b4a34b479edb3fd8491e63b1: Generated new TSK 0
I20260927 03:59:59.383085 27368 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.185.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab.
Entry for principal kudu/127.15.185.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab.
I20260927 04:00:00.083811 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.185.84:59725/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.65:0
--local_ip_for_outbound_sockets=127.15.185.65
--webserver_interface=127.15.185.65
--webserver_port=0
--tserver_master_addrs=127.15.185.126:36961
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab
--principal=kudu/127.15.185.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:35299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:00:00.191924 27400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:00:00.192202 27400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:00:00.192243 27400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:00:00.193442 27400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:00:00.193511 27400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:00:00.193557 27400 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.65
I20260927 04:00:00.195465 27400 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:35299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.15.185.84:59725/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab
--principal=kudu/127.15.185.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.185.65
--webserver_port=0
--tserver_master_addrs=127.15.185.126:36961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.65
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:00:00.195735 27400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:00:00.195976 27400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:00:00.202286 27406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:00:00.202411 27408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:00:00.202517 27400 server_base.cc:1034] running on GCE node
W20260927 04:00:00.202275 27405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:00:00.202833 27400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:00:00.203166 27400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:00:00.204347 27400 hybrid_clock.cc:648] HybridClock initialized: now 1790481600204325 us; error 39 us; skew 500 ppm
Sep 27 04:00:00 dist-test-slave-glgz krb5kdc[26843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481600, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:00:00.207036 27400 init.cc:377] Logged in from keytab as kudu/127.15.185.65@KRBTEST.COM (short username kudu)
I20260927 04:00:00.207623 27400 webserver.cc:460] Webserver started at http://127.15.185.65:41929/ using document root <none> and password file <none>
I20260927 04:00:00.207901 27400 fs_manager.cc:359] Metadata directory not provided
I20260927 04:00:00.207976 27400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:00:00.208170 27400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 04:00:00 dist-test-slave-glgz krb5kdc[26843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481600, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.65@KRBTEST.COM for HTTP/127.15.185.84@KRBTEST.COM
I20260927 04:00:00.221331 27400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ts-0/data/instance:
uuid: "1b3b353b4bc0433c949f19bb46f36e0f"
format_stamp: "Formatted at 2026-09-27 04:00:00 on dist-test-slave-glgz"
server_key: "9d55aaf9b4ec4be525829b53c0dc9206"
server_key_iv: "e4a14f2edc9ff87ec91a2ba2382d75b6"
server_key_version: "kuduclusterkey@0"
I20260927 04:00:00.221827 27400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ts-0/wal/instance:
uuid: "1b3b353b4bc0433c949f19bb46f36e0f"
format_stamp: "Formatted at 2026-09-27 04:00:00 on dist-test-slave-glgz"
server_key: "9d55aaf9b4ec4be525829b53c0dc9206"
server_key_iv: "e4a14f2edc9ff87ec91a2ba2382d75b6"
server_key_version: "kuduclusterkey@0"
I20260927 04:00:00.223800 27400 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260927 04:00:00.243309 27415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:00:00.243580 27400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260927 04:00:00.243676 27400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ts-0/wal
uuid: "1b3b353b4bc0433c949f19bb46f36e0f"
format_stamp: "Formatted at 2026-09-27 04:00:00 on dist-test-slave-glgz"
server_key: "9d55aaf9b4ec4be525829b53c0dc9206"
server_key_iv: "e4a14f2edc9ff87ec91a2ba2382d75b6"
server_key_version: "kuduclusterkey@0"
I20260927 04:00:00.243798 27400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:00:00.271862 27400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:00:00.274535 27400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:00:00.274765 27400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:00:00.275067 27400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:00:00.275460 27400 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:00:00.275532 27400 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:00:00.275586 27400 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:00:00.275629 27400 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:00:00.283092 27400 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.65:38865
I20260927 04:00:00.283599 27400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
I20260927 04:00:00.284019 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 27400
I20260927 04:00:00.284134 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ts-0/wal/instance
I20260927 04:00:00.284340 16101 external_mini_cluster.cc:1428] Setting key b77f80d39ec661cf0fa8b179eaf6b82c
WARNING: no policy specified for kudu/127.15.185.66@KRBTEST.COM; defaulting to no policy
I20260927 04:00:00.300565 27530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.65:38865 every 8 connection(s)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.66@KRBTEST.COM" created.
Sep 27 04:00:00 dist-test-slave-glgz krb5kdc[26843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481600, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.65@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 04:00:00.320317 27534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:00.299590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:36961 (local address 127.15.185.65:50545)
0927 04:00:00.301045 (+ 1455us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:00.301089 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 04:00:00.301809 (+ 720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:00:00.302107 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:00:00.302111 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:00:00.302300 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:00:00.302505 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:00.302508 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:00.303424 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:00.303425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:00.304856 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:00.304859 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:00.305445 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:00.305446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:00.305605 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:00.306284 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:00:00.306297 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:00:00.309686 (+ 3389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:00:00.317783 (+ 8097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:00:00.317800 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:00:00.317803 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:00:00.318183 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:00:00.318563 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:00:00.318564 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:00:00.318564 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:00:00.318722 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:00:00.319490 (+ 768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:00:00.319499 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:00:00.319739 (+ 240us) client_negotiation.cc:769] Sending connection context
0927 04:00:00.319844 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 04:00:00.320024 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1408,"thread_start_us":104,"threads_started":1}
I20260927 04:00:00.320822 27535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:00.299927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:50545 (local address 127.15.185.126:36961)
0927 04:00:00.301845 (+ 1918us) server_negotiation.cc:206] Beginning negotiation
0927 04:00:00.301848 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:00:00.301859 (+ 11us) server_negotiation.cc:407] Connection header received
0927 04:00:00.301934 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:00:00.301935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:00:00.301947 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:00:00.302023 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:00:00.302615 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:00.303327 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:00.304958 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:00.305303 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:00.305369 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:00.309807 (+ 4438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:00:00.309809 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:00:00.309810 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:00:00.309819 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:00:00.317547 (+ 7728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:00.318398 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:00.318399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:00.318399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:00.318482 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:00.319223 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:00.319224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:00.319225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:00.319394 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:00:00.320572 (+ 1178us) server_negotiation.cc:1035] Waiting for connection context
0927 04:00:00.320619 (+ 47us) server_negotiation.cc:299] Negotiation successful
0927 04:00:00.320722 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1886,"thread_start_us":69,"threads_started":1}
I20260927 04:00:00.321430 27531 heartbeater.cc:347] Connected to a master server at 127.15.185.126:36961
I20260927 04:00:00.321565 27531 heartbeater.cc:464] Registering TS with master...
I20260927 04:00:00.321797 27531 heartbeater.cc:511] Master 127.15.185.126:36961 requested a full tablet report, sending...
I20260927 04:00:00.322384 27319 ts_manager.cc:194] Registered new tserver with Master: 1b3b353b4bc0433c949f19bb46f36e0f (127.15.185.65:38865)
I20260927 04:00:00.323354 27319 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.185.65@KRBTEST.COM'} at 127.15.185.65:50545
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab.
Entry for principal kudu/127.15.185.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab.
I20260927 04:00:00.330330 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.185.84:59725/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.66:0
--local_ip_for_outbound_sockets=127.15.185.66
--webserver_interface=127.15.185.66
--webserver_port=0
--tserver_master_addrs=127.15.185.126:36961
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab
--principal=kudu/127.15.185.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:35299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:00:00.442461 27537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:00:00.442724 27537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:00:00.442823 27537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:00:00.444204 27537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:00:00.444281 27537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:00:00.444334 27537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.66
I20260927 04:00:00.446107 27537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:35299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.15.185.84:59725/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab
--principal=kudu/127.15.185.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.185.66
--webserver_port=0
--tserver_master_addrs=127.15.185.126:36961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.66
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:00:00.446391 27537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:00:00.446686 27537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:00:00.453009 27547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:00:00.453083 27545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:00:00.453372 27544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:00:00.454314 27537 server_base.cc:1034] running on GCE node
I20260927 04:00:00.454535 27537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:00:00.454840 27537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:00:00.456000 27537 hybrid_clock.cc:648] HybridClock initialized: now 1790481600455979 us; error 39 us; skew 500 ppm
Sep 27 04:00:00 dist-test-slave-glgz krb5kdc[26843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481600, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:00:00.458812 27537 init.cc:377] Logged in from keytab as kudu/127.15.185.66@KRBTEST.COM (short username kudu)
I20260927 04:00:00.459357 27537 webserver.cc:460] Webserver started at http://127.15.185.66:41099/ using document root <none> and password file <none>
I20260927 04:00:00.459636 27537 fs_manager.cc:359] Metadata directory not provided
I20260927 04:00:00.459713 27537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:00:00.459914 27537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 04:00:00 dist-test-slave-glgz krb5kdc[26843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481600, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.66@KRBTEST.COM for HTTP/127.15.185.84@KRBTEST.COM
I20260927 04:00:00.472781 27537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ts-1/data/instance:
uuid: "5580f2f396e249938479feaa7bc3480b"
format_stamp: "Formatted at 2026-09-27 04:00:00 on dist-test-slave-glgz"
server_key: "cfc84c147e1ffcfa1adbb7c2caefaf75"
server_key_iv: "1ba458696be077be93f70e97f52447dd"
server_key_version: "kuduclusterkey@0"
I20260927 04:00:00.473907 27537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ts-1/wal/instance:
uuid: "5580f2f396e249938479feaa7bc3480b"
format_stamp: "Formatted at 2026-09-27 04:00:00 on dist-test-slave-glgz"
server_key: "cfc84c147e1ffcfa1adbb7c2caefaf75"
server_key_iv: "1ba458696be077be93f70e97f52447dd"
server_key_version: "kuduclusterkey@0"
I20260927 04:00:00.476295 27537 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
Sep 27 04:00:00 dist-test-slave-glgz krb5kdc[26843](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790481600, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:00:00.500385 27554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:00:00.500779 27537 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.002s sys 0.000s
I20260927 04:00:00.500901 27537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ts-1/wal
uuid: "5580f2f396e249938479feaa7bc3480b"
format_stamp: "Formatted at 2026-09-27 04:00:00 on dist-test-slave-glgz"
server_key: "cfc84c147e1ffcfa1adbb7c2caefaf75"
server_key_iv: "1ba458696be077be93f70e97f52447dd"
server_key_version: "kuduclusterkey@0"
I20260927 04:00:00.501013 27537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:00:00.524974 27537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:00:00.527786 27537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:00:00.528029 27537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:00:00.528321 27537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:00:00.528867 27537 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:00:00.528970 27537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:00:00.529048 27537 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:00:00.529109 27537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:00:00.537822 27537 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.66:43819
I20260927 04:00:00.538388 27537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
I20260927 04:00:00.541191 27667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.66:43819 every 8 connection(s)
I20260927 04:00:00.546953 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 27537
I20260927 04:00:00.547149 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ts-1/wal/instance
I20260927 04:00:00.547374 16101 external_mini_cluster.cc:1428] Setting key e5e2663e5435d6d030f19de8e0c5855f
Sep 27 04:00:00 dist-test-slave-glgz krb5kdc[26843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481600, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.66@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 04:00:00.554736 27670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:00.540520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:36961 (local address 127.15.185.66:38861)
0927 04:00:00.541602 (+ 1082us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:00.541643 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 04:00:00.542291 (+ 648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:00:00.542628 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:00:00.542632 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:00:00.542868 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:00:00.543166 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:00.543171 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:00.544172 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:00.544173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:00.545502 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:00.545505 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:00.546037 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:00.546038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:00.546173 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:00.547670 (+ 1497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:00:00.547682 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:00:00.551157 (+ 3475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:00:00.553021 (+ 1864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:00:00.553027 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:00:00.553030 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:00:00.553351 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:00:00.553563 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:00:00.553564 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:00:00.553564 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:00:00.553699 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:00:00.554027 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:00:00.554032 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:00:00.554203 (+ 171us) client_negotiation.cc:769] Sending connection context
0927 04:00:00.554322 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 04:00:00.554481 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1044,"thread_start_us":79,"threads_started":1}
I20260927 04:00:00.555269 27535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:00.540788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:38861 (local address 127.15.185.126:36961)
0927 04:00:00.540837 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 04:00:00.540840 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:00:00.541730 (+ 890us) server_negotiation.cc:407] Connection header received
0927 04:00:00.542403 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:00:00.542404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:00:00.542420 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:00:00.542451 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:00:00.543287 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:00.544075 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:00.545604 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:00.545917 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:00.545957 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:00.551277 (+ 5320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:00:00.551279 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:00:00.551279 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:00:00.551287 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:00:00.552927 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:00.553436 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:00.553437 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:00.553438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:00.553497 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:00.553801 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:00.553802 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:00.553802 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:00.553943 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:00:00.554003 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 04:00:00.555078 (+ 1075us) server_negotiation.cc:299] Negotiation successful
0927 04:00:00.555169 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 04:00:00.555961 27668 heartbeater.cc:347] Connected to a master server at 127.15.185.126:36961
I20260927 04:00:00.556057 27668 heartbeater.cc:464] Registering TS with master...
I20260927 04:00:00.556267 27668 heartbeater.cc:511] Master 127.15.185.126:36961 requested a full tablet report, sending...
I20260927 04:00:00.556634 27319 ts_manager.cc:194] Registered new tserver with Master: 5580f2f396e249938479feaa7bc3480b (127.15.185.66:43819)
I20260927 04:00:00.557128 27319 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.185.66@KRBTEST.COM'} at 127.15.185.66:38861
WARNING: no policy specified for kudu/127.15.185.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab.
Entry for principal kudu/127.15.185.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab.
I20260927 04:00:00.587224 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.185.84:59725/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.67:0
--local_ip_for_outbound_sockets=127.15.185.67
--webserver_interface=127.15.185.67
--webserver_port=0
--tserver_master_addrs=127.15.185.126:36961
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab
--principal=kudu/127.15.185.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:35299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:00:00.683282 27674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:00:00.683524 27674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:00:00.683568 27674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:00:00.684924 27674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:00:00.685032 27674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:00:00.685093 27674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.67
I20260927 04:00:00.687331 27674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:35299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.15.185.84:59725/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab
--principal=kudu/127.15.185.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.185.67
--webserver_port=0
--tserver_master_addrs=127.15.185.126:36961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.67
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:00:00.687723 27674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:00:00.688043 27674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:00:00.693393 27682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:00:00.693511 27679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:00:00.693387 27680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:00:00.694634 27674 server_base.cc:1034] running on GCE node
I20260927 04:00:00.694886 27674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:00:00.695180 27674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:00:00.702597 27674 hybrid_clock.cc:648] HybridClock initialized: now 1790481600702559 us; error 41 us; skew 500 ppm
Sep 27 04:00:00 dist-test-slave-glgz krb5kdc[26843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481600, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:00:00.706413 27674 init.cc:377] Logged in from keytab as kudu/127.15.185.67@KRBTEST.COM (short username kudu)
I20260927 04:00:00.707079 27674 webserver.cc:460] Webserver started at http://127.15.185.67:45109/ using document root <none> and password file <none>
I20260927 04:00:00.707376 27674 fs_manager.cc:359] Metadata directory not provided
I20260927 04:00:00.707456 27674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:00:00.707654 27674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 04:00:00 dist-test-slave-glgz krb5kdc[26843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481600, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.67@KRBTEST.COM for HTTP/127.15.185.84@KRBTEST.COM
I20260927 04:00:00.721153 27674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ts-2/data/instance:
uuid: "210c141b32854a309a3c9e3cd141e40f"
format_stamp: "Formatted at 2026-09-27 04:00:00 on dist-test-slave-glgz"
server_key: "2166e4671b952b9d5347ffd09f6e47e3"
server_key_iv: "9a3a2b8c5cb13ddf7b17fc174b1f3d95"
server_key_version: "kuduclusterkey@0"
I20260927 04:00:00.721745 27674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ts-2/wal/instance:
uuid: "210c141b32854a309a3c9e3cd141e40f"
format_stamp: "Formatted at 2026-09-27 04:00:00 on dist-test-slave-glgz"
server_key: "2166e4671b952b9d5347ffd09f6e47e3"
server_key_iv: "9a3a2b8c5cb13ddf7b17fc174b1f3d95"
server_key_version: "kuduclusterkey@0"
I20260927 04:00:00.724928 27674 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 04:00:00.739929 27689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:00:00.740197 27674 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:00:00.740304 27674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ts-2/wal
uuid: "210c141b32854a309a3c9e3cd141e40f"
format_stamp: "Formatted at 2026-09-27 04:00:00 on dist-test-slave-glgz"
server_key: "2166e4671b952b9d5347ffd09f6e47e3"
server_key_iv: "9a3a2b8c5cb13ddf7b17fc174b1f3d95"
server_key_version: "kuduclusterkey@0"
I20260927 04:00:00.740425 27674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:00:00.776444 27674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:00:00.778937 27674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:00:00.779158 27674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:00:00.779439 27674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:00:00.779845 27674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:00:00.779898 27674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:00:00.779932 27674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:00:00.779958 27674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:00:00.788514 27674 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.67:45987
I20260927 04:00:00.788618 27802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.67:45987 every 8 connection(s)
I20260927 04:00:00.788954 27674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
Sep 27 04:00:00 dist-test-slave-glgz krb5kdc[26843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481600, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.67@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 04:00:00.796505 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 27674
I20260927 04:00:00.796639 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ts-2/wal/instance
I20260927 04:00:00.796849 16101 external_mini_cluster.cc:1428] Setting key 0b4cce4d31bf01b7796dd5fab5446dc9
I20260927 04:00:00.800168 27535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:00.790255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:35725 (local address 127.15.185.126:36961)
0927 04:00:00.790323 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 04:00:00.790325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:00:00.790568 (+ 243us) server_negotiation.cc:407] Connection header received
0927 04:00:00.791199 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:00:00.791200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:00:00.791212 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:00:00.791268 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:00:00.791871 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:00.792380 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:00.793437 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:00.793654 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:00.793759 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:00.796362 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:00:00.796364 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:00:00.796365 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:00:00.796373 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:00:00.798429 (+ 2056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:00.799004 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:00.799005 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:00.799006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:00.799130 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:00.799461 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:00.799462 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:00.799462 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:00.799600 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:00:00.799679 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 04:00:00.799983 (+ 304us) server_negotiation.cc:299] Negotiation successful
0927 04:00:00.800072 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 04:00:00.800530 27805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:00.790231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:36961 (local address 127.15.185.67:35725)
0927 04:00:00.790435 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:00.790476 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 04:00:00.791030 (+ 554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:00:00.791358 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:00:00.791362 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:00:00.791555 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:00:00.791761 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:00.791765 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:00.792483 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:00.792484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:00.793332 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:00.793335 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:00.793771 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:00.793772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:00.793905 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:00.794507 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:00:00.794516 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:00:00.796189 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:00:00.798522 (+ 2333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:00:00.798532 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:00:00.798535 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:00:00.798911 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:00:00.799233 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:00:00.799234 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:00:00.799235 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:00:00.799380 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:00:00.799670 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:00:00.799677 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:00:00.799843 (+ 166us) client_negotiation.cc:769] Sending connection context
0927 04:00:00.799963 (+ 120us) client_negotiation.cc:240] Negotiation successful
0927 04:00:00.800133 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":132,"threads_started":1}
I20260927 04:00:00.801090 27803 heartbeater.cc:347] Connected to a master server at 127.15.185.126:36961
I20260927 04:00:00.801210 27803 heartbeater.cc:464] Registering TS with master...
I20260927 04:00:00.801419 27803 heartbeater.cc:511] Master 127.15.185.126:36961 requested a full tablet report, sending...
I20260927 04:00:00.801827 27319 ts_manager.cc:194] Registered new tserver with Master: 210c141b32854a309a3c9e3cd141e40f (127.15.185.67:45987)
I20260927 04:00:00.802253 27319 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.185.67@KRBTEST.COM'} at 127.15.185.67:35725
I20260927 04:00:00.809101 16101 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 04:00:00.818975 27535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:00.811128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54426 (local address 127.15.185.126:36961)
0927 04:00:00.811176 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 04:00:00.811178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:00:00.811184 (+ 6us) server_negotiation.cc:407] Connection header received
0927 04:00:00.811203 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:00:00.811204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:00:00.811213 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:00:00.811277 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:00:00.811923 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:00.812721 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:00.813864 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:00.814166 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:00.814209 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:00.815886 (+ 1677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:00:00.815887 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:00:00.815888 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:00:00.815895 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:00:00.817636 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:00.818040 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:00.818041 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:00.818042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:00.818094 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:00.818281 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:00.818282 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:00.818282 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:00.818492 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:00:00.818571 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 04:00:00.818800 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 04:00:00.818882 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 04:00:00.822031 27319 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54426:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 04:00:00.824046 27319 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 04:00:00.833400 27817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:00.827442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49440 (local address 127.15.185.66:43819)
0927 04:00:00.828021 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0927 04:00:00.828026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:00:00.828362 (+ 336us) server_negotiation.cc:407] Connection header received
0927 04:00:00.828618 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:00:00.828621 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:00:00.828640 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:00:00.828725 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:00:00.829117 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:00.830256 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:00.831899 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:00.832833 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:00.832885 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:00.832965 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 04:00:00.833256 (+ 291us) server_negotiation.cc:299] Negotiation successful
0927 04:00:00.833302 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":80,"threads_started":1}
I20260927 04:00:00.833665 27819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:00.827308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.66:43819 (local address 127.0.0.1:49440)
0927 04:00:00.828248 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:00.828291 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 04:00:00.828477 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:00:00.828738 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:00:00.828742 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:00:00.828744 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:00:00.828819 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:00.828820 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:00.830365 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:00.830365 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:00.831771 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:00.831773 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:00.833031 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:00.833032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:00.833124 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:00.833139 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 04:00:00.833516 (+ 377us) client_negotiation.cc:240] Negotiation successful
0927 04:00:00.833590 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":910,"thread_start_us":82,"threads_started":1}
I20260927 04:00:00.834455 27602 tablet_service.cc:1467] Processing CreateTablet for tablet e75119927fdc4db392c1946b38e8664e (DEFAULT_TABLE table=test-table [id=d26e0882b1884af2bdc16b5f0cc413af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:00:00.834873 27602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e75119927fdc4db392c1946b38e8664e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:00:00.836716 27818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:00.827551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.67:45987 (local address 127.0.0.1:60486)
0927 04:00:00.828606 (+ 1055us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:00.828615 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 04:00:00.828656 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:00:00.831156 (+ 2500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:00:00.831158 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:00:00.831160 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:00:00.831226 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:00.831227 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:00.834147 (+ 2920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:00.834148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:00.835302 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:00.835306 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:00.836449 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:00.836451 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:00.836528 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:00.836533 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 04:00:00.836588 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 04:00:00.836630 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1040,"thread_start_us":78,"threads_started":1}
I20260927 04:00:00.837143 27820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:00.827876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60486 (local address 127.15.185.67:45987)
0927 04:00:00.828097 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 04:00:00.828101 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:00:00.828888 (+ 787us) server_negotiation.cc:407] Connection header received
0927 04:00:00.828932 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:00:00.828935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:00:00.828950 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:00:00.829018 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:00:00.831324 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:00.834005 (+ 2681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:00.835428 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:00.836236 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:00.836314 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:00.836410 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 04:00:00.837015 (+ 605us) server_negotiation.cc:299] Negotiation successful
0927 04:00:00.837063 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":84,"threads_started":1}
I20260927 04:00:00.838276 27737 tablet_service.cc:1467] Processing CreateTablet for tablet e75119927fdc4db392c1946b38e8664e (DEFAULT_TABLE table=test-table [id=d26e0882b1884af2bdc16b5f0cc413af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:00:00.838641 27737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e75119927fdc4db392c1946b38e8664e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:00:00.841293 27816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:00.827884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.65:38865 (local address 127.0.0.1:44388)
0927 04:00:00.832740 (+ 4856us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:00.832751 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 04:00:00.832813 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:00:00.833104 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:00:00.833105 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:00:00.833107 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:00:00.833169 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:00.833171 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:00.838810 (+ 5639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:00.838811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:00.839995 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:00.839998 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:00.841041 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:00.841042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:00.841117 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:00.841122 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 04:00:00.841172 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 04:00:00.841215 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4834,"thread_start_us":75,"threads_started":1}
I20260927 04:00:00.841754 27821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:00.828019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44388 (local address 127.15.185.65:38865)
0927 04:00:00.828434 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0927 04:00:00.828438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:00:00.832880 (+ 4442us) server_negotiation.cc:407] Connection header received
0927 04:00:00.832921 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:00:00.832924 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:00:00.832942 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:00:00.833027 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:00:00.833246 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:00.838628 (+ 5382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:00.840100 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:00.840870 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:00.840944 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:00.841004 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 04:00:00.841610 (+ 606us) server_negotiation.cc:299] Negotiation successful
0927 04:00:00.841658 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":52,"threads_started":1}
I20260927 04:00:00.842852 27465 tablet_service.cc:1467] Processing CreateTablet for tablet e75119927fdc4db392c1946b38e8664e (DEFAULT_TABLE table=test-table [id=d26e0882b1884af2bdc16b5f0cc413af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:00:00.843230 27465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e75119927fdc4db392c1946b38e8664e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:00:00.844482 27822 tablet_bootstrap.cc:492] T e75119927fdc4db392c1946b38e8664e P 210c141b32854a309a3c9e3cd141e40f: Bootstrap starting.
I20260927 04:00:00.845583 27822 tablet_bootstrap.cc:654] T e75119927fdc4db392c1946b38e8664e P 210c141b32854a309a3c9e3cd141e40f: Neither blocks nor log segments found. Creating new log.
I20260927 04:00:00.846305 27822 log.cc:824] T e75119927fdc4db392c1946b38e8664e P 210c141b32854a309a3c9e3cd141e40f: Log is configured to *not* fsync() on all Append() calls
I20260927 04:00:00.846661 27823 tablet_bootstrap.cc:492] T e75119927fdc4db392c1946b38e8664e P 5580f2f396e249938479feaa7bc3480b: Bootstrap starting.
I20260927 04:00:00.847393 27825 tablet_bootstrap.cc:492] T e75119927fdc4db392c1946b38e8664e P 1b3b353b4bc0433c949f19bb46f36e0f: Bootstrap starting.
I20260927 04:00:00.847848 27823 tablet_bootstrap.cc:654] T e75119927fdc4db392c1946b38e8664e P 5580f2f396e249938479feaa7bc3480b: Neither blocks nor log segments found. Creating new log.
I20260927 04:00:00.848460 27825 tablet_bootstrap.cc:654] T e75119927fdc4db392c1946b38e8664e P 1b3b353b4bc0433c949f19bb46f36e0f: Neither blocks nor log segments found. Creating new log.
I20260927 04:00:00.848539 27822 tablet_bootstrap.cc:492] T e75119927fdc4db392c1946b38e8664e P 210c141b32854a309a3c9e3cd141e40f: No bootstrap required, opened a new log
I20260927 04:00:00.848609 27822 ts_tablet_manager.cc:1397] T e75119927fdc4db392c1946b38e8664e P 210c141b32854a309a3c9e3cd141e40f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 04:00:00.849018 27825 log.cc:824] T e75119927fdc4db392c1946b38e8664e P 1b3b353b4bc0433c949f19bb46f36e0f: Log is configured to *not* fsync() on all Append() calls
I20260927 04:00:00.848145 27823 log.cc:824] T e75119927fdc4db392c1946b38e8664e P 5580f2f396e249938479feaa7bc3480b: Log is configured to *not* fsync() on all Append() calls
I20260927 04:00:00.850165 27825 tablet_bootstrap.cc:492] T e75119927fdc4db392c1946b38e8664e P 1b3b353b4bc0433c949f19bb46f36e0f: No bootstrap required, opened a new log
I20260927 04:00:00.850231 27825 ts_tablet_manager.cc:1397] T e75119927fdc4db392c1946b38e8664e P 1b3b353b4bc0433c949f19bb46f36e0f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 04:00:00.851078 27822 raft_consensus.cc:348] T e75119927fdc4db392c1946b38e8664e P 210c141b32854a309a3c9e3cd141e40f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "210c141b32854a309a3c9e3cd141e40f" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 45987 } } peers { permanent_uuid: "1b3b353b4bc0433c949f19bb46f36e0f" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 38865 } } peers { permanent_uuid: "5580f2f396e249938479feaa7bc3480b" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 43819 } }
I20260927 04:00:00.851241 27822 raft_consensus.cc:374] T e75119927fdc4db392c1946b38e8664e P 210c141b32854a309a3c9e3cd141e40f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:00:00.851271 27822 raft_consensus.cc:729] T e75119927fdc4db392c1946b38e8664e P 210c141b32854a309a3c9e3cd141e40f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 210c141b32854a309a3c9e3cd141e40f, State: Initialized, Role: FOLLOWER
I20260927 04:00:00.851402 27822 consensus_queue.cc:260] T e75119927fdc4db392c1946b38e8664e P 210c141b32854a309a3c9e3cd141e40f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "210c141b32854a309a3c9e3cd141e40f" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 45987 } } peers { permanent_uuid: "1b3b353b4bc0433c949f19bb46f36e0f" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 38865 } } peers { permanent_uuid: "5580f2f396e249938479feaa7bc3480b" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 43819 } }
I20260927 04:00:00.851650 27822 ts_tablet_manager.cc:1428] T e75119927fdc4db392c1946b38e8664e P 210c141b32854a309a3c9e3cd141e40f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 04:00:00.851776 27803 heartbeater.cc:503] Master 127.15.185.126:36961 was elected leader, sending a full tablet report...
I20260927 04:00:00.852931 27825 raft_consensus.cc:348] T e75119927fdc4db392c1946b38e8664e P 1b3b353b4bc0433c949f19bb46f36e0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "210c141b32854a309a3c9e3cd141e40f" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 45987 } } peers { permanent_uuid: "1b3b353b4bc0433c949f19bb46f36e0f" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 38865 } } peers { permanent_uuid: "5580f2f396e249938479feaa7bc3480b" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 43819 } }
I20260927 04:00:00.853082 27825 raft_consensus.cc:374] T e75119927fdc4db392c1946b38e8664e P 1b3b353b4bc0433c949f19bb46f36e0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:00:00.853116 27825 raft_consensus.cc:729] T e75119927fdc4db392c1946b38e8664e P 1b3b353b4bc0433c949f19bb46f36e0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b3b353b4bc0433c949f19bb46f36e0f, State: Initialized, Role: FOLLOWER
I20260927 04:00:00.853134 27823 tablet_bootstrap.cc:492] T e75119927fdc4db392c1946b38e8664e P 5580f2f396e249938479feaa7bc3480b: No bootstrap required, opened a new log
I20260927 04:00:00.853193 27823 ts_tablet_manager.cc:1397] T e75119927fdc4db392c1946b38e8664e P 5580f2f396e249938479feaa7bc3480b: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260927 04:00:00.853504 27825 consensus_queue.cc:260] T e75119927fdc4db392c1946b38e8664e P 1b3b353b4bc0433c949f19bb46f36e0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "210c141b32854a309a3c9e3cd141e40f" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 45987 } } peers { permanent_uuid: "1b3b353b4bc0433c949f19bb46f36e0f" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 38865 } } peers { permanent_uuid: "5580f2f396e249938479feaa7bc3480b" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 43819 } }
I20260927 04:00:00.853909 27825 ts_tablet_manager.cc:1428] T e75119927fdc4db392c1946b38e8664e P 1b3b353b4bc0433c949f19bb46f36e0f: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260927 04:00:00.854066 27531 heartbeater.cc:503] Master 127.15.185.126:36961 was elected leader, sending a full tablet report...
I20260927 04:00:00.855675 27823 raft_consensus.cc:348] T e75119927fdc4db392c1946b38e8664e P 5580f2f396e249938479feaa7bc3480b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "210c141b32854a309a3c9e3cd141e40f" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 45987 } } peers { permanent_uuid: "1b3b353b4bc0433c949f19bb46f36e0f" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 38865 } } peers { permanent_uuid: "5580f2f396e249938479feaa7bc3480b" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 43819 } }
I20260927 04:00:00.855865 27823 raft_consensus.cc:374] T e75119927fdc4db392c1946b38e8664e P 5580f2f396e249938479feaa7bc3480b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:00:00.855902 27823 raft_consensus.cc:729] T e75119927fdc4db392c1946b38e8664e P 5580f2f396e249938479feaa7bc3480b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5580f2f396e249938479feaa7bc3480b, State: Initialized, Role: FOLLOWER
I20260927 04:00:00.856215 27823 consensus_queue.cc:260] T e75119927fdc4db392c1946b38e8664e P 5580f2f396e249938479feaa7bc3480b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "210c141b32854a309a3c9e3cd141e40f" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 45987 } } peers { permanent_uuid: "1b3b353b4bc0433c949f19bb46f36e0f" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 38865 } } peers { permanent_uuid: "5580f2f396e249938479feaa7bc3480b" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 43819 } }
I20260927 04:00:00.856484 27823 ts_tablet_manager.cc:1428] T e75119927fdc4db392c1946b38e8664e P 5580f2f396e249938479feaa7bc3480b: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 04:00:00.856593 27668 heartbeater.cc:503] Master 127.15.185.126:36961 was elected leader, sending a full tablet report...
I20260927 04:00:00.924563 27830 raft_consensus.cc:482] T e75119927fdc4db392c1946b38e8664e P 5580f2f396e249938479feaa7bc3480b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:00:00.924769 27830 raft_consensus.cc:504] T e75119927fdc4db392c1946b38e8664e P 5580f2f396e249938479feaa7bc3480b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "210c141b32854a309a3c9e3cd141e40f" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 45987 } } peers { permanent_uuid: "1b3b353b4bc0433c949f19bb46f36e0f" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 38865 } } peers { permanent_uuid: "5580f2f396e249938479feaa7bc3480b" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 43819 } }
I20260927 04:00:00.926415 27830 leader_election.cc:288] T e75119927fdc4db392c1946b38e8664e P 5580f2f396e249938479feaa7bc3480b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 210c141b32854a309a3c9e3cd141e40f (127.15.185.67:45987), 1b3b353b4bc0433c949f19bb46f36e0f (127.15.185.65:38865)
I20260927 04:00:00.929769 27670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:00.925369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.67:45987 (local address 127.15.185.66:49763)
0927 04:00:00.925475 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:00.925487 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 04:00:00.925529 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:00:00.925813 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:00:00.925814 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:00:00.925816 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:00:00.925890 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:00.925892 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:00.926892 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:00.926893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:00.928611 (+ 1718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:00.928614 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:00.929498 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:00.929499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:00.929584 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:00.929588 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 04:00:00.929641 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 04:00:00.929688 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260927 04:00:00.930078 27820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:00.925609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:49763 (local address 127.15.185.67:45987)
0927 04:00:00.925650 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 04:00:00.925652 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:00:00.925659 (+ 7us) server_negotiation.cc:407] Connection header received
0927 04:00:00.925688 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:00:00.925689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:00:00.925698 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:00:00.925745 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:00:00.925979 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:00.926717 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:00.928810 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:00.929372 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:00.929453 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:00.929464 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 04:00:00.929969 (+ 505us) server_negotiation.cc:299] Negotiation successful
0927 04:00:00.930011 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 04:00:00.930356 27757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e75119927fdc4db392c1946b38e8664e" candidate_uuid: "5580f2f396e249938479feaa7bc3480b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "210c141b32854a309a3c9e3cd141e40f" is_pre_election: true
I20260927 04:00:00.930521 27757 raft_consensus.cc:2454] T e75119927fdc4db392c1946b38e8664e P 210c141b32854a309a3c9e3cd141e40f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5580f2f396e249938479feaa7bc3480b in term 0.
I20260927 04:00:00.930765 27556 leader_election.cc:302] T e75119927fdc4db392c1946b38e8664e P 5580f2f396e249938479feaa7bc3480b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 210c141b32854a309a3c9e3cd141e40f, 5580f2f396e249938479feaa7bc3480b; no voters:
I20260927 04:00:00.930913 27830 raft_consensus.cc:2781] T e75119927fdc4db392c1946b38e8664e P 5580f2f396e249938479feaa7bc3480b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 04:00:00.930951 27830 raft_consensus.cc:482] T e75119927fdc4db392c1946b38e8664e P 5580f2f396e249938479feaa7bc3480b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 04:00:00.930972 27830 raft_consensus.cc:3037] T e75119927fdc4db392c1946b38e8664e P 5580f2f396e249938479feaa7bc3480b [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:00:00.932008 27830 raft_consensus.cc:504] T e75119927fdc4db392c1946b38e8664e P 5580f2f396e249938479feaa7bc3480b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "210c141b32854a309a3c9e3cd141e40f" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 45987 } } peers { permanent_uuid: "1b3b353b4bc0433c949f19bb46f36e0f" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 38865 } } peers { permanent_uuid: "5580f2f396e249938479feaa7bc3480b" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 43819 } }
I20260927 04:00:00.932163 27830 leader_election.cc:288] T e75119927fdc4db392c1946b38e8664e P 5580f2f396e249938479feaa7bc3480b [CANDIDATE]: Term 1 election: Requested vote from peers 210c141b32854a309a3c9e3cd141e40f (127.15.185.67:45987), 1b3b353b4bc0433c949f19bb46f36e0f (127.15.185.65:38865)
I20260927 04:00:00.932367 27757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e75119927fdc4db392c1946b38e8664e" candidate_uuid: "5580f2f396e249938479feaa7bc3480b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "210c141b32854a309a3c9e3cd141e40f"
I20260927 04:00:00.932457 27757 raft_consensus.cc:3037] T e75119927fdc4db392c1946b38e8664e P 210c141b32854a309a3c9e3cd141e40f [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:00:00.933354 27757 raft_consensus.cc:2454] T e75119927fdc4db392c1946b38e8664e P 210c141b32854a309a3c9e3cd141e40f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5580f2f396e249938479feaa7bc3480b in term 1.
I20260927 04:00:00.933543 27556 leader_election.cc:302] T e75119927fdc4db392c1946b38e8664e P 5580f2f396e249938479feaa7bc3480b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 210c141b32854a309a3c9e3cd141e40f, 5580f2f396e249938479feaa7bc3480b; no voters:
I20260927 04:00:00.933631 27830 raft_consensus.cc:2781] T e75119927fdc4db392c1946b38e8664e P 5580f2f396e249938479feaa7bc3480b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:00:00.933815 27830 raft_consensus.cc:686] T e75119927fdc4db392c1946b38e8664e P 5580f2f396e249938479feaa7bc3480b [term 1 LEADER]: Becoming Leader. State: Replica: 5580f2f396e249938479feaa7bc3480b, State: Running, Role: LEADER
I20260927 04:00:00.933920 27830 consensus_queue.cc:237] T e75119927fdc4db392c1946b38e8664e P 5580f2f396e249938479feaa7bc3480b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "210c141b32854a309a3c9e3cd141e40f" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 45987 } } peers { permanent_uuid: "1b3b353b4bc0433c949f19bb46f36e0f" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 38865 } } peers { permanent_uuid: "5580f2f396e249938479feaa7bc3480b" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 43819 } }
I20260927 04:00:00.934793 27319 catalog_manager.cc:5495] T e75119927fdc4db392c1946b38e8664e P 5580f2f396e249938479feaa7bc3480b reported cstate change: term changed from 0 to 1, leader changed from <none> to 5580f2f396e249938479feaa7bc3480b (127.15.185.66). New cstate: current_term: 1 leader_uuid: "5580f2f396e249938479feaa7bc3480b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "210c141b32854a309a3c9e3cd141e40f" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 45987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b3b353b4bc0433c949f19bb46f36e0f" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 38865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5580f2f396e249938479feaa7bc3480b" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 43819 } health_report { overall_health: HEALTHY } } }
I20260927 04:00:00.935487 27821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:00.926347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:46731 (local address 127.15.185.65:38865)
0927 04:00:00.926391 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 04:00:00.926393 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:00:00.927473 (+ 1080us) server_negotiation.cc:407] Connection header received
0927 04:00:00.927581 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:00:00.927582 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:00:00.927592 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:00:00.927622 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:00:00.927852 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:00.928584 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:00.930659 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:00.931300 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:00.931352 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:00.931360 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 04:00:00.935349 (+ 3989us) server_negotiation.cc:299] Negotiation successful
0927 04:00:00.935414 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 04:00:00.935763 27831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:00.926119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.65:38865 (local address 127.15.185.66:46731)
0927 04:00:00.927398 (+ 1279us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:00.927411 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 04:00:00.927502 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:00:00.927685 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:00:00.927686 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:00:00.927687 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:00:00.927767 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:00.927769 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:00.928909 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:00.928909 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:00.930421 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:00.930423 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:00.931559 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:00.931560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:00.935233 (+ 3673us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:00.935241 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 04:00:00.935613 (+ 372us) client_negotiation.cc:240] Negotiation successful
0927 04:00:00.935667 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1256,"thread_start_us":119,"threads_started":1}
I20260927 04:00:00.936195 27485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e75119927fdc4db392c1946b38e8664e" candidate_uuid: "5580f2f396e249938479feaa7bc3480b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3b353b4bc0433c949f19bb46f36e0f" is_pre_election: true
I20260927 04:00:00.936384 27485 raft_consensus.cc:2454] T e75119927fdc4db392c1946b38e8664e P 1b3b353b4bc0433c949f19bb46f36e0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5580f2f396e249938479feaa7bc3480b in term 0.
I20260927 04:00:00.936506 27484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e75119927fdc4db392c1946b38e8664e" candidate_uuid: "5580f2f396e249938479feaa7bc3480b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b3b353b4bc0433c949f19bb46f36e0f"
I20260927 04:00:00.936569 27484 raft_consensus.cc:3037] T e75119927fdc4db392c1946b38e8664e P 1b3b353b4bc0433c949f19bb46f36e0f [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:00:00.937613 27484 raft_consensus.cc:2454] T e75119927fdc4db392c1946b38e8664e P 1b3b353b4bc0433c949f19bb46f36e0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5580f2f396e249938479feaa7bc3480b in term 1.
I20260927 04:00:00.955929 27817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:00.952599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49456 (local address 127.15.185.66:43819)
0927 04:00:00.952653 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 04:00:00.952655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:00:00.952665 (+ 10us) server_negotiation.cc:407] Connection header received
0927 04:00:00.952708 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:00:00.952709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:00:00.952751 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:00:00.952836 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:00:00.953005 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:00.953755 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:00.954881 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:00.955219 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:00.955293 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:00.955504 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:00:00.955654 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:00:00.955742 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 04:00:00.955818 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 04:00:00.955844 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 04:00:00.958406 27484 raft_consensus.cc:1261] T e75119927fdc4db392c1946b38e8664e P 1b3b353b4bc0433c949f19bb46f36e0f [term 1 FOLLOWER]: Refusing update from remote peer 5580f2f396e249938479feaa7bc3480b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:00:00.958803 27757 raft_consensus.cc:1261] T e75119927fdc4db392c1946b38e8664e P 210c141b32854a309a3c9e3cd141e40f [term 1 FOLLOWER]: Refusing update from remote peer 5580f2f396e249938479feaa7bc3480b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:00:00.959280 27830 consensus_queue.cc:1035] T e75119927fdc4db392c1946b38e8664e P 5580f2f396e249938479feaa7bc3480b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b3b353b4bc0433c949f19bb46f36e0f" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 38865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:00:00.959465 27830 consensus_queue.cc:1035] T e75119927fdc4db392c1946b38e8664e P 5580f2f396e249938479feaa7bc3480b [LEADER]: Connected to new peer: Peer: permanent_uuid: "210c141b32854a309a3c9e3cd141e40f" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 45987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:00:00.964073 27836 mvcc.cc:205] Tried to move back new op lower bound from 7333812637521567744 to 7333812637425942528. Current Snapshot: MvccSnapshot[applied={T|T < 7333812637521567744}]
I20260927 04:00:00.964578 27835 mvcc.cc:205] Tried to move back new op lower bound from 7333812637521567744 to 7333812637425942528. Current Snapshot: MvccSnapshot[applied={T|T < 7333812637521567744}]
I20260927 04:00:00.966372 27838 mvcc.cc:205] Tried to move back new op lower bound from 7333812637521567744 to 7333812637425942528. Current Snapshot: MvccSnapshot[applied={T|T < 7333812637521567744 or (T in {7333812637521567744})}]
I20260927 04:00:00.969419 27319 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54426:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:00:00.969519 27319 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54426:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:00:00.970315 27319 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 27c553f8b4a34b479edb3fd8491e63b1: Sending DeleteTablet for 3 replicas of tablet e75119927fdc4db392c1946b38e8664e
I20260927 04:00:00.971169 27602 tablet_service.cc:1514] Processing DeleteTablet for tablet e75119927fdc4db392c1946b38e8664e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:00:00 UTC) from {username='kudu', principal='kudu/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:49440
I20260927 04:00:00.971211 27737 tablet_service.cc:1514] Processing DeleteTablet for tablet e75119927fdc4db392c1946b38e8664e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:00:00 UTC) from {username='kudu', principal='kudu/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:60486
I20260927 04:00:00.971450 27465 tablet_service.cc:1514] Processing DeleteTablet for tablet e75119927fdc4db392c1946b38e8664e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:00:00 UTC) from {username='kudu', principal='kudu/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:44388
I20260927 04:00:00.975158 27846 tablet_replica.cc:330] T e75119927fdc4db392c1946b38e8664e P 210c141b32854a309a3c9e3cd141e40f: stopping tablet replica
I20260927 04:00:00.975315 27846 raft_consensus.cc:2229] T e75119927fdc4db392c1946b38e8664e P 210c141b32854a309a3c9e3cd141e40f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:00:00.975447 27846 raft_consensus.cc:2258] T e75119927fdc4db392c1946b38e8664e P 210c141b32854a309a3c9e3cd141e40f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:00:00.975653 27847 tablet_replica.cc:330] T e75119927fdc4db392c1946b38e8664e P 1b3b353b4bc0433c949f19bb46f36e0f: stopping tablet replica
I20260927 04:00:00.975762 27847 raft_consensus.cc:2229] T e75119927fdc4db392c1946b38e8664e P 1b3b353b4bc0433c949f19bb46f36e0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:00:00.975869 27847 raft_consensus.cc:2258] T e75119927fdc4db392c1946b38e8664e P 1b3b353b4bc0433c949f19bb46f36e0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:00:00.976377 27846 ts_tablet_manager.cc:1905] T e75119927fdc4db392c1946b38e8664e P 210c141b32854a309a3c9e3cd141e40f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:00:00.978116 27846 ts_tablet_manager.cc:1918] T e75119927fdc4db392c1946b38e8664e P 210c141b32854a309a3c9e3cd141e40f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:00:00.978191 27846 log.cc:1196] T e75119927fdc4db392c1946b38e8664e P 210c141b32854a309a3c9e3cd141e40f: Deleting WAL directory at /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ts-2/wal/wals/e75119927fdc4db392c1946b38e8664e
I20260927 04:00:00.978793 27846 ts_tablet_manager.cc:1939] T e75119927fdc4db392c1946b38e8664e P 210c141b32854a309a3c9e3cd141e40f: Deleting consensus metadata
I20260927 04:00:00.979279 27304 catalog_manager.cc:4841] TS 210c141b32854a309a3c9e3cd141e40f (127.15.185.67:45987): tablet e75119927fdc4db392c1946b38e8664e (table test-table [id=d26e0882b1884af2bdc16b5f0cc413af]) successfully deleted
I20260927 04:00:00.979420 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 27400
I20260927 04:00:00.983970 27845 tablet_replica.cc:330] T e75119927fdc4db392c1946b38e8664e P 5580f2f396e249938479feaa7bc3480b: stopping tablet replica
I20260927 04:00:00.984119 27845 raft_consensus.cc:2229] T e75119927fdc4db392c1946b38e8664e P 5580f2f396e249938479feaa7bc3480b [term 1 LEADER]: Raft consensus shutting down.
I20260927 04:00:00.984283 27845 raft_consensus.cc:2258] T e75119927fdc4db392c1946b38e8664e P 5580f2f396e249938479feaa7bc3480b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:00:00.984839 27845 ts_tablet_manager.cc:1905] T e75119927fdc4db392c1946b38e8664e P 5580f2f396e249938479feaa7bc3480b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:00:00.986696 27845 ts_tablet_manager.cc:1918] T e75119927fdc4db392c1946b38e8664e P 5580f2f396e249938479feaa7bc3480b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:00:00.995121 27845 log.cc:1196] T e75119927fdc4db392c1946b38e8664e P 5580f2f396e249938479feaa7bc3480b: Deleting WAL directory at /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481498018407-16101-0/minicluster-data/ts-1/wal/wals/e75119927fdc4db392c1946b38e8664e
I20260927 04:00:00.995630 27845 ts_tablet_manager.cc:1939] T e75119927fdc4db392c1946b38e8664e P 5580f2f396e249938479feaa7bc3480b: Deleting consensus metadata
W20260927 04:00:00.995891 27303 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.185.65:38865 (error 108)
I20260927 04:00:00.997071 27306 catalog_manager.cc:4841] TS 5580f2f396e249938479feaa7bc3480b (127.15.185.66:43819): tablet e75119927fdc4db392c1946b38e8664e (table test-table [id=d26e0882b1884af2bdc16b5f0cc413af]) successfully deleted
I20260927 04:00:00.997392 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 27537
I20260927 04:00:00.998486 27816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:00.998360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.65:38865 (local address 127.0.0.1:44394)
0927 04:00:00.998411 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:00.998446 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.65:38865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 04:00:00.998616 27303 catalog_manager.cc:4584] TS 1b3b353b4bc0433c949f19bb46f36e0f (127.15.185.65:38865): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e75119927fdc4db392c1946b38e8664e: Network error: Client connection negotiation failed: client connection to 127.15.185.65:38865: connect: Connection refused (error 111)
Sep 27 04:00:00 dist-test-slave-glgz krb5kdc[26843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481600, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.126@KRBTEST.COM for HTTP/127.15.185.84@KRBTEST.COM
I20260927 04:00:01.003999 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 27674
I20260927 04:00:01.014449 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 27287
I20260927 04:00:01.022535 16101 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 04:00:01.367918 16101 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 04:00:01.368077 16101 mini_ranger.cc:66] Stopping Ranger...
I20260927 04:00:01.713320 16101 mini_ranger.cc:68] Stopped Ranger
2026-09-27 04:00:01.713 UTC [26873] LOG: received smart shutdown request
2026-09-27 04:00:01.717 UTC [26873] LOG: background worker "logical replication launcher" (PID 26882) exited with exit code 1
2026-09-27 04:00:01.717 UTC [26877] LOG: shutting down
2026-09-27 04:00:01.908 UTC [26873] LOG: database system is shut down
2026-09-27T04:00:01Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47180 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:00:02 dist-test-slave-glgz krb5kdc[27857](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:00:02 dist-test-slave-glgz krb5kdc[27857](info): set up 2 sockets
Sep 27 04:00:02 dist-test-slave-glgz krb5kdc[27857](info): commencing operation
krb5kdc: starting...
W20260927 04:00:03.063715 16101 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Sep 27 04:00:03 dist-test-slave-glgz krb5kdc[27857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:00:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:00:03Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.15.185.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.15.185.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.15.185.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/krb5kdc/rangeradmin_127.15.185.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.15.185.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/krb5kdc/rangeradmin_127.15.185.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.15.185.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.15.185.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.15.185.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/krb5kdc/rangerlookup_127.15.185.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.15.185.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/krb5kdc/rangerlookup_127.15.185.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.15.185.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.15.185.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.15.185.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/krb5kdc/HTTP_127.15.185.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.15.185.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/krb5kdc/HTTP_127.15.185.84@KRBTEST.COM.keytab.
I20260927 04:00:03.243036 16101 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskTmmW3q/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/postgres -l logfile start
2026-09-27 04:00:05.909 UTC [27887] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 04:00:05.909 UTC [27887] LOG: listening on IPv4 address "127.15.185.84", port 36163
2026-09-27 04:00:05.914 UTC [27887] LOG: listening on Unix socket "/tmp/.s.PGSQL.36163"
2026-09-27 04:00:05.953 UTC [27890] LOG: database system was shut down at 2026-09-27 04:00:04 UTC
2026-09-27 04:00:05.957 UTC [27887] LOG: database system is ready to accept connections
I20260927 04:00:06.923372 16101 mini_postgres.cc:96] Postgres bound to 36163
2026-09-27 04:00:06.929 UTC [27898] FATAL: database "slave" does not exist
127.15.185.84:36163 - accepting connections
I20260927 04:00:06.930107 16101 mini_ranger.cc:161] Starting Ranger...
I20260927 04:00:06.945600 16101 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 04:00:07.495498 16101 mini_ranger.cc:87] Created ranger Postgres database
I20260927 04:00:07.495608 16101 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ranger-admin
2026-09-27 04:00:07,774 [I] DB FLAVOR :POSTGRES
2026-09-27 04:00:07,775 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 04:00:07,775 [I] Checking connection..
2026-09-27 04:00:07,775 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTmmW3q/build/release/bin/postgresql.jar:/tmp/dist-test-taskTmmW3q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.185.84:36163/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 04:00:08,044 [I] Checking connection passed.
2026-09-27 04:00:08,044 [I] --------- Verifying version history table ---------
2026-09-27 04:00:08,045 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTmmW3q/build/release/bin/postgresql.jar:/tmp/dist-test-taskTmmW3q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.185.84:36163/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 04:00:08,299 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 04:00:08,300 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTmmW3q/build/release/bin/postgresql.jar:/tmp/dist-test-taskTmmW3q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.185.84:36163/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 04:00:08,566 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 04:00:08,566 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 04:00:08,566 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTmmW3q/build/release/bin/postgresql.jar:/tmp/dist-test-taskTmmW3q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.185.84:36163/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskTmmW3q/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 04:00:08.811 UTC [27962] WARNING: there is no transaction in progress
2026-09-27 04:00:08,818 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 04:00:08,819 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTmmW3q/build/release/bin/postgresql.jar:/tmp/dist-test-taskTmmW3q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.185.84:36163/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 04:00:09,062 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 04:00:09,062 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 04:00:09,062 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTmmW3q/build/release/bin/postgresql.jar:/tmp/dist-test-taskTmmW3q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.185.84:36163/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 04:00:09,300 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTmmW3q/build/release/bin/postgresql.jar:/tmp/dist-test-taskTmmW3q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.185.84:36163/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 04:00:09,561 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTmmW3q/build/release/bin/postgresql.jar:/tmp/dist-test-taskTmmW3q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.185.84:36163/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-glgz.c.gcp-upstream.internal','N') ;"
2026-09-27 04:00:09,797 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTmmW3q/build/release/bin/postgresql.jar:/tmp/dist-test-taskTmmW3q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.185.84:36163/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 04:00:10,028 [I] Table x_portal_user does not exist in database ranger
2026-09-27 04:00:10,028 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTmmW3q/build/release/bin/postgresql.jar:/tmp/dist-test-taskTmmW3q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.185.84:36163/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 04:00:10,280 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 04:00:10,280 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTmmW3q/build/release/bin/postgresql.jar:/tmp/dist-test-taskTmmW3q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.185.84:36163/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 04:00:10,509 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTmmW3q/build/release/bin/postgresql.jar:/tmp/dist-test-taskTmmW3q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.185.84:36163/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 04:00:10,745 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 04:00:10,745 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTmmW3q/build/release/bin/postgresql.jar:/tmp/dist-test-taskTmmW3q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.185.84:36163/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskTmmW3q/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 04:00:11.775 UTC [28070] WARNING: there is no transaction in progress
2026-09-27 04:00:11.793 UTC [28070] WARNING: there is no transaction in progress
2026-09-27 04:00:11.811 UTC [28070] WARNING: there is no transaction in progress
2026-09-27 04:00:11.829 UTC [28070] WARNING: there is no transaction in progress
2026-09-27 04:00:11.846 UTC [28070] WARNING: there is no transaction in progress
2026-09-27 04:00:11.868 UTC [28070] WARNING: there is no transaction in progress
2026-09-27 04:00:11.927 UTC [28070] WARNING: there is no transaction in progress
2026-09-27 04:00:11.935 UTC [28070] WARNING: there is no transaction in progress
2026-09-27 04:00:11.951 UTC [28070] WARNING: there is no transaction in progress
2026-09-27 04:00:11.962 UTC [28070] WARNING: there is no transaction in progress
2026-09-27 04:00:11.973 UTC [28070] WARNING: there is no transaction in progress
2026-09-27 04:00:11.988 UTC [28070] WARNING: there is no transaction in progress
2026-09-27 04:00:11.998 UTC [28070] WARNING: there is no transaction in progress
2026-09-27 04:00:12.005 UTC [28070] WARNING: there is no transaction in progress
2026-09-27 04:00:12.013 UTC [28070] WARNING: there is no transaction in progress
2026-09-27 04:00:12.020 UTC [28070] WARNING: there is no transaction in progress
2026-09-27 04:00:12,749 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 04:00:12,750 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTmmW3q/build/release/bin/postgresql.jar:/tmp/dist-test-taskTmmW3q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.185.84:36163/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 04:00:12,992 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 04:00:12,992 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTmmW3q/build/release/bin/postgresql.jar:/tmp/dist-test-taskTmmW3q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.185.84:36163/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 04:00:13,265 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 04:00:13,265 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTmmW3q/build/release/bin/postgresql.jar:/tmp/dist-test-taskTmmW3q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.185.84:36163/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 04:00:13,510 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 04:00:13,510 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTmmW3q/build/release/bin/postgresql.jar:/tmp/dist-test-taskTmmW3q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.185.84:36163/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 04:00:13,745 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTmmW3q/build/release/bin/postgresql.jar:/tmp/dist-test-taskTmmW3q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.185.84:36163/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 04:00:13,983 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTmmW3q/build/release/bin/postgresql.jar:/tmp/dist-test-taskTmmW3q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.185.84:36163/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-glgz.c.gcp-upstream.internal';"
2026-09-27 04:00:14,216 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 04:00:14,217 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTmmW3q/build/release/bin/postgresql.jar:/tmp/dist-test-taskTmmW3q/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.185.84:36163/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 04:00:14,462 [I] DB_PATCHES have already been applied
I20260927 04:00:14.468087 16101 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ranger-admin:/tmp/dist-test-taskTmmW3q/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskTmmW3q/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskTmmW3q/build/release/bin/postgresql.jar:/tmp/dist-test-taskTmmW3q/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 04:00:14.468184 16101 mini_ranger.cc:193] Using host: 127.15.185.84
I20260927 04:00:14.471462 16101 mini_ranger.cc:227] Ranger admin URL: http://127.15.185.84:41143
Sep 27, 2026 4:00:14 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 4:00:14 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 4:00:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 4:00:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 4:00:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 4:00:15 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 4:00:15 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 04:00:15.589843 16101 mini_ranger.cc:160] Time spent starting Ranger: real 8.660s user 0.000s sys 0.009s
Sep 27, 2026 4:00:34 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 4:00:35 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 27, 2026 4:00:35 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 4:00:35 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 4:00:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 4:00:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 4:00:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 4:00:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 4:00:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 4:00:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 4:00:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 4:00:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 4:00:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 4:00:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 4:00:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 4:00:35 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 4:00:35 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 4:00:36 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 04:00:36.585377 16101 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.15.185.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.15.185.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.15.185.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/krb5kdc/rangerkms_127.15.185.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.15.185.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/krb5kdc/rangerkms_127.15.185.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.15.185.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.15.185.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.15.185.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/krb5kdc/HTTP_127.15.185.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.15.185.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/krb5kdc/HTTP_127.15.185.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 04:00:36 dist-test-slave-glgz krb5kdc[27857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481636, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 04:00:36.667395 16101 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 04:00:36.676847 16101 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 04:00:37.553788 16101 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 04:00:37.553884 16101 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ranger-kms
I20260927 04:00:37.553905 16101 mini_ranger_kms.cc:227] Using postgres at 127.15.185.84:36163
2026-09-27 04:00:37,721 [I] DB FLAVOR :POSTGRES
2026-09-27 04:00:37,721 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 04:00:37,721 [I] Checking connection
2026-09-27 04:00:37,721 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTmmW3q/build/release/bin/postgresql.jar:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.185.84:36163/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 04:00:37,976 [I] connection success
2026-09-27 04:00:37,976 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 04:00:37,976 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 04:00:37,976 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTmmW3q/build/release/bin/postgresql.jar:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.185.84:36163/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 04:00:38,278 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 04:00:38,278 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 04:00:38,278 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 04:00:38,278 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTmmW3q/build/release/bin/postgresql.jar:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.185.84:36163/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 04:00:38,531 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 04:00:38.997697 16101 mini_ranger_kms.cc:310] Created kms service
I20260927 04:00:39.187472 16101 mini_ranger_kms.cc:326] Created kudu user
I20260927 04:00:39.237437 16101 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 04:00:39 dist-test-slave-glgz krb5kdc[27857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481636, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.15.185.84@KRBTEST.COM
I20260927 04:00:39.630952 16101 mini_ranger_kms.cc:384] Added ranger policy
I20260927 04:00:39.631209 16101 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ranger-kms:/tmp/dist-test-taskTmmW3q/build/release/bin/postgresql.jar:/tmp/dist-test-taskTmmW3q/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskTmmW3q/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 04:00:39.631242 16101 mini_ranger_kms.cc:240] Using host: 127.15.185.84
I20260927 04:00:39.634639 16101 mini_ranger_kms.cc:276] Ranger KMS PID: 28250
I20260927 04:00:39.634723 16101 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.15.185.84:46557
Sep 27, 2026 4:00:40 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 4:00:40 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 4:00:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 4:00:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 4:00:41 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 4:00:41 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 04:00:41.947105 16101 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.280s user 0.004s sys 0.146s
I20260927 04:00:41.947214 16101 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 04:00:41.947265 16101 mini_ranger_kms.cc:401] 127.15.185.84:46557/kms/v1/keys
Sep 27 04:00:42 dist-test-slave-glgz krb5kdc[27857](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790481642, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.185.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 04:00:42 dist-test-slave-glgz krb5kdc[27857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481642, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.185.84@KRBTEST.COM for HTTP/127.15.185.84@KRBTEST.COM
Sep 27 04:00:43 dist-test-slave-glgz krb5kdc[27857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481642, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.185.84@KRBTEST.COM for HTTP/127.15.185.84@KRBTEST.COM
Sep 27 04:00:45 dist-test-slave-glgz krb5kdc[27857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.15.185.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab.
Entry for principal kudu/127.15.185.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab.
I20260927 04:00:45.334241 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:38903
--webserver_interface=127.15.185.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:46163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.185.126:38903
--ranger_config_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.185.84:46557/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:00:45.416592 28301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:00:45.416807 28301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:00:45.416829 28301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:00:45.418035 28301 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:00:45.418074 28301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:00:45.418088 28301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:00:45.418102 28301 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:00:45.418113 28301 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:00:45.419682 28301 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:46163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/master-0/wal
--ranger_kms_url=127.15.185.84:46557/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.15.185.126:38903
--ranger_config_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:38903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:00:45.419931 28301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:00:45.420152 28301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 04:00:45.425308 28301 server_base.cc:1034] running on GCE node
W20260927 04:00:45.425448 28307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:00:45.425477 28306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:00:45.425523 28309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:00:45.425820 28301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:00:45.426072 28301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:00:45.427204 28301 hybrid_clock.cc:648] HybridClock initialized: now 1790481645427193 us; error 28 us; skew 500 ppm
Sep 27 04:00:45 dist-test-slave-glgz krb5kdc[27857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481645, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:00:45.434571 28301 init.cc:377] Logged in from keytab as kudu/127.15.185.126@KRBTEST.COM (short username kudu)
I20260927 04:00:45.435091 28301 webserver.cc:460] Webserver started at http://127.15.185.126:46047/ using document root <none> and password file <none>
I20260927 04:00:45.435278 28301 fs_manager.cc:359] Metadata directory not provided
I20260927 04:00:45.435315 28301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:00:45.435405 28301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 04:00:45 dist-test-slave-glgz krb5kdc[27857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481645, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.126@KRBTEST.COM for HTTP/127.15.185.84@KRBTEST.COM
I20260927 04:00:45.486850 28301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/master-0/data/instance:
uuid: "aff4fb0709a14bfbbc2e9060c6d43b72"
format_stamp: "Formatted at 2026-09-27 04:00:45 on dist-test-slave-glgz"
server_key: "0f43b0da6edd7820ba2a4e5b585a6c95"
server_key_iv: "1ea03daf82bde83bc6945a013afcf56f"
server_key_version: "kuduclusterkey@0"
I20260927 04:00:45.487289 28301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/master-0/wal/instance:
uuid: "aff4fb0709a14bfbbc2e9060c6d43b72"
format_stamp: "Formatted at 2026-09-27 04:00:45 on dist-test-slave-glgz"
server_key: "0f43b0da6edd7820ba2a4e5b585a6c95"
server_key_iv: "1ea03daf82bde83bc6945a013afcf56f"
server_key_version: "kuduclusterkey@0"
I20260927 04:00:45.489015 28301 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 04:00:45.501587 28316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:00:45.504534 28301 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 04:00:45.504676 28301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/master-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/master-0/wal
uuid: "aff4fb0709a14bfbbc2e9060c6d43b72"
format_stamp: "Formatted at 2026-09-27 04:00:45 on dist-test-slave-glgz"
server_key: "0f43b0da6edd7820ba2a4e5b585a6c95"
server_key_iv: "1ea03daf82bde83bc6945a013afcf56f"
server_key_version: "kuduclusterkey@0"
I20260927 04:00:45.504791 28301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:00:45.522326 28301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:00:45.524457 28301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:00:45.524616 28301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:00:45.529516 28301 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.126:38903
I20260927 04:00:45.529608 28368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.126:38903 every 8 connection(s)
I20260927 04:00:45.529973 28301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
I20260927 04:00:45.530684 28369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:00:45.533000 28369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aff4fb0709a14bfbbc2e9060c6d43b72: Bootstrap starting.
I20260927 04:00:45.533591 28369 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aff4fb0709a14bfbbc2e9060c6d43b72: Neither blocks nor log segments found. Creating new log.
I20260927 04:00:45.533877 28369 log.cc:824] T 00000000000000000000000000000000 P aff4fb0709a14bfbbc2e9060c6d43b72: Log is configured to *not* fsync() on all Append() calls
I20260927 04:00:45.534591 28369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aff4fb0709a14bfbbc2e9060c6d43b72: No bootstrap required, opened a new log
I20260927 04:00:45.536731 28369 raft_consensus.cc:348] T 00000000000000000000000000000000 P aff4fb0709a14bfbbc2e9060c6d43b72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff4fb0709a14bfbbc2e9060c6d43b72" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 38903 } }
I20260927 04:00:45.536909 28369 raft_consensus.cc:374] T 00000000000000000000000000000000 P aff4fb0709a14bfbbc2e9060c6d43b72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:00:45.536974 28369 raft_consensus.cc:729] T 00000000000000000000000000000000 P aff4fb0709a14bfbbc2e9060c6d43b72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aff4fb0709a14bfbbc2e9060c6d43b72, State: Initialized, Role: FOLLOWER
I20260927 04:00:45.537132 28369 consensus_queue.cc:260] T 00000000000000000000000000000000 P aff4fb0709a14bfbbc2e9060c6d43b72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff4fb0709a14bfbbc2e9060c6d43b72" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 38903 } }
I20260927 04:00:45.537206 28369 raft_consensus.cc:388] T 00000000000000000000000000000000 P aff4fb0709a14bfbbc2e9060c6d43b72 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:00:45.537249 28369 raft_consensus.cc:482] T 00000000000000000000000000000000 P aff4fb0709a14bfbbc2e9060c6d43b72 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:00:45.537297 28369 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aff4fb0709a14bfbbc2e9060c6d43b72 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:00:45.537824 28369 raft_consensus.cc:504] T 00000000000000000000000000000000 P aff4fb0709a14bfbbc2e9060c6d43b72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff4fb0709a14bfbbc2e9060c6d43b72" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 38903 } }
I20260927 04:00:45.537943 28369 leader_election.cc:302] T 00000000000000000000000000000000 P aff4fb0709a14bfbbc2e9060c6d43b72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aff4fb0709a14bfbbc2e9060c6d43b72; no voters:
I20260927 04:00:45.538120 28369 leader_election.cc:288] T 00000000000000000000000000000000 P aff4fb0709a14bfbbc2e9060c6d43b72 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:00:45.538164 28372 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aff4fb0709a14bfbbc2e9060c6d43b72 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:00:45.538357 28369 sys_catalog.cc:564] T 00000000000000000000000000000000 P aff4fb0709a14bfbbc2e9060c6d43b72 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:00:45.538380 28372 raft_consensus.cc:686] T 00000000000000000000000000000000 P aff4fb0709a14bfbbc2e9060c6d43b72 [term 1 LEADER]: Becoming Leader. State: Replica: aff4fb0709a14bfbbc2e9060c6d43b72, State: Running, Role: LEADER
I20260927 04:00:45.538460 28372 consensus_queue.cc:237] T 00000000000000000000000000000000 P aff4fb0709a14bfbbc2e9060c6d43b72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff4fb0709a14bfbbc2e9060c6d43b72" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 38903 } }
I20260927 04:00:45.538844 28372 sys_catalog.cc:455] T 00000000000000000000000000000000 P aff4fb0709a14bfbbc2e9060c6d43b72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aff4fb0709a14bfbbc2e9060c6d43b72. Latest consensus state: current_term: 1 leader_uuid: "aff4fb0709a14bfbbc2e9060c6d43b72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff4fb0709a14bfbbc2e9060c6d43b72" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 38903 } } }
I20260927 04:00:45.539074 28372 sys_catalog.cc:458] T 00000000000000000000000000000000 P aff4fb0709a14bfbbc2e9060c6d43b72 [sys.catalog]: This master's current role is: LEADER
I20260927 04:00:45.538993 28373 sys_catalog.cc:455] T 00000000000000000000000000000000 P aff4fb0709a14bfbbc2e9060c6d43b72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aff4fb0709a14bfbbc2e9060c6d43b72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff4fb0709a14bfbbc2e9060c6d43b72" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 38903 } } }
I20260927 04:00:45.539247 28373 sys_catalog.cc:458] T 00000000000000000000000000000000 P aff4fb0709a14bfbbc2e9060c6d43b72 [sys.catalog]: This master's current role is: LEADER
I20260927 04:00:45.539916 28369 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 04:00:45.540091 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 28301
I20260927 04:00:45.540202 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/master-0/wal/instance
I20260927 04:00:45.540303 28381 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:00:45.540421 16101 external_mini_cluster.cc:1428] Setting key 25699af044f7520a90006471727046bf
I20260927 04:00:45.540903 28381 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:00:45.545958 28381 catalog_manager.cc:1347] Generated new cluster ID: a7b4ab978a7643e9967a6f35bf499c48
I20260927 04:00:45.546096 28381 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 04:00:45 dist-test-slave-glgz krb5kdc[27857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 04:00:45.554852 28381 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:00:45.555707 28381 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:00:45.556196 28391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:45.541083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52630 (local address 127.15.185.126:38903)
0927 04:00:45.543314 (+ 2231us) server_negotiation.cc:206] Beginning negotiation
0927 04:00:45.543326 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 04:00:45.543399 (+ 73us) server_negotiation.cc:407] Connection header received
0927 04:00:45.543748 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:00:45.543760 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:00:45.543884 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:00:45.544082 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:00:45.544932 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:45.546716 (+ 1784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:45.547939 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:45.548600 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:45.549029 (+ 429us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:45.551946 (+ 2917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:00:45.551950 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:00:45.551951 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:00:45.551963 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:00:45.554007 (+ 2044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:45.554436 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:45.554439 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:45.554439 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:45.554539 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:45.554754 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:45.554755 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:45.554755 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:45.555117 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:00:45.555728 (+ 611us) server_negotiation.cc:1035] Waiting for connection context
0927 04:00:45.555803 (+ 75us) server_negotiation.cc:299] Negotiation successful
0927 04:00:45.555960 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2037,"thread_start_us":66,"threads_started":1}
I20260927 04:00:45.561549 28381 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aff4fb0709a14bfbbc2e9060c6d43b72: Generated new TSK 0
I20260927 04:00:45.561766 28381 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.185.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab.
Entry for principal kudu/127.15.185.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab.
I20260927 04:00:46.210697 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.185.84:46557/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.65:0
--local_ip_for_outbound_sockets=127.15.185.65
--webserver_interface=127.15.185.65
--webserver_port=0
--tserver_master_addrs=127.15.185.126:38903
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab
--principal=kudu/127.15.185.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:46163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:00:46.322342 28414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:00:46.322652 28414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:00:46.322690 28414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:00:46.324501 28414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:00:46.324569 28414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:00:46.324611 28414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.65
I20260927 04:00:46.326844 28414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:46163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.15.185.84:46557/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab
--principal=kudu/127.15.185.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.185.65
--webserver_port=0
--tserver_master_addrs=127.15.185.126:38903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.65
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:00:46.327147 28414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:00:46.327471 28414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:00:46.334443 28422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:00:46.334547 28420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:00:46.334456 28419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:00:46.335574 28414 server_base.cc:1034] running on GCE node
I20260927 04:00:46.335937 28414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:00:46.336222 28414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:00:46.337386 28414 hybrid_clock.cc:648] HybridClock initialized: now 1790481646337360 us; error 43 us; skew 500 ppm
Sep 27 04:00:46 dist-test-slave-glgz krb5kdc[27857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481646, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:00:46.340586 28414 init.cc:377] Logged in from keytab as kudu/127.15.185.65@KRBTEST.COM (short username kudu)
I20260927 04:00:46.341151 28414 webserver.cc:460] Webserver started at http://127.15.185.65:36205/ using document root <none> and password file <none>
I20260927 04:00:46.341491 28414 fs_manager.cc:359] Metadata directory not provided
I20260927 04:00:46.341559 28414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:00:46.341763 28414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 04:00:46 dist-test-slave-glgz krb5kdc[27857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481646, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.65@KRBTEST.COM for HTTP/127.15.185.84@KRBTEST.COM
I20260927 04:00:46.358307 28414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-0/data/instance:
uuid: "9069f7895034488b852820a86b6ac57c"
format_stamp: "Formatted at 2026-09-27 04:00:46 on dist-test-slave-glgz"
server_key: "977779661d056c4e8e055e954a0e1c15"
server_key_iv: "afadfee72a603d90cc6e89b0e2da8d72"
server_key_version: "kuduclusterkey@0"
I20260927 04:00:46.358888 28414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-0/wal/instance:
uuid: "9069f7895034488b852820a86b6ac57c"
format_stamp: "Formatted at 2026-09-27 04:00:46 on dist-test-slave-glgz"
server_key: "977779661d056c4e8e055e954a0e1c15"
server_key_iv: "afadfee72a603d90cc6e89b0e2da8d72"
server_key_version: "kuduclusterkey@0"
I20260927 04:00:46.361083 28414 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260927 04:00:46.379191 28430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:00:46.379462 28414 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s
I20260927 04:00:46.379566 28414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-0/wal
uuid: "9069f7895034488b852820a86b6ac57c"
format_stamp: "Formatted at 2026-09-27 04:00:46 on dist-test-slave-glgz"
server_key: "977779661d056c4e8e055e954a0e1c15"
server_key_iv: "afadfee72a603d90cc6e89b0e2da8d72"
server_key_version: "kuduclusterkey@0"
I20260927 04:00:46.379659 28414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:00:46.401125 28414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:00:46.403842 28414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:00:46.404019 28414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:00:46.404317 28414 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:00:46.404722 28414 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:00:46.404778 28414 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:00:46.404812 28414 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:00:46.404870 28414 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:00:46.412675 28414 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.65:36243
I20260927 04:00:46.412747 28544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.65:36243 every 8 connection(s)
I20260927 04:00:46.413158 28414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
I20260927 04:00:46.418730 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 28414
I20260927 04:00:46.418853 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-0/wal/instance
I20260927 04:00:46.419071 16101 external_mini_cluster.cc:1428] Setting key bd5d534c372f4664a42f74bf6024363f
Sep 27 04:00:46 dist-test-slave-glgz krb5kdc[27857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481646, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.65@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 04:00:46.425630 28547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:46.414310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:38903 (local address 127.15.185.65:44977)
0927 04:00:46.414597 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:46.414645 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 04:00:46.415367 (+ 722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:00:46.415666 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:00:46.415671 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:00:46.415942 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:00:46.416260 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:46.416268 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:46.416997 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:46.416998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:46.418361 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:46.418364 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:46.418972 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:46.418973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:46.419177 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:46.419834 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:00:46.419845 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:00:46.421606 (+ 1761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:00:46.423821 (+ 2215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:00:46.423830 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:00:46.423832 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:00:46.424162 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:00:46.424388 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:00:46.424389 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:00:46.424389 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:00:46.424528 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:00:46.424855 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:00:46.424862 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:00:46.425047 (+ 185us) client_negotiation.cc:769] Sending connection context
0927 04:00:46.425185 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 04:00:46.425370 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":90,"threads_started":1}
I20260927 04:00:46.426247 28548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:46.414341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:44977 (local address 127.15.185.126:38903)
0927 04:00:46.415180 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0927 04:00:46.415183 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:00:46.415196 (+ 13us) server_negotiation.cc:407] Connection header received
0927 04:00:46.415487 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:00:46.415488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:00:46.415500 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:00:46.415579 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:00:46.416379 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:46.416893 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:46.418479 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:46.418806 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:46.418883 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:46.421725 (+ 2842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:00:46.421727 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:00:46.421728 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:00:46.421736 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:00:46.423720 (+ 1984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:46.424253 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:46.424255 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:46.424255 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:46.424321 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:46.424605 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:46.424605 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:46.424606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:46.424758 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:00:46.424825 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 04:00:46.426042 (+ 1217us) server_negotiation.cc:299] Negotiation successful
0927 04:00:46.426136 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":806,"thread_start_us":92,"threads_started":1}
I20260927 04:00:46.427716 28545 heartbeater.cc:347] Connected to a master server at 127.15.185.126:38903
I20260927 04:00:46.427822 28545 heartbeater.cc:464] Registering TS with master...
I20260927 04:00:46.428032 28545 heartbeater.cc:511] Master 127.15.185.126:38903 requested a full tablet report, sending...
I20260927 04:00:46.428983 28333 ts_manager.cc:194] Registered new tserver with Master: 9069f7895034488b852820a86b6ac57c (127.15.185.65:36243)
I20260927 04:00:46.429924 28333 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.185.65@KRBTEST.COM'} at 127.15.185.65:44977
WARNING: no policy specified for kudu/127.15.185.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab.
Entry for principal kudu/127.15.185.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab.
I20260927 04:00:46.456784 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.185.84:46557/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.66:0
--local_ip_for_outbound_sockets=127.15.185.66
--webserver_interface=127.15.185.66
--webserver_port=0
--tserver_master_addrs=127.15.185.126:38903
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab
--principal=kudu/127.15.185.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:46163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 04:00:46 dist-test-slave-glgz krb5kdc[27857](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790481646, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 04:00:46.577878 28551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:00:46.578260 28551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:00:46.578342 28551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:00:46.580248 28551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:00:46.580356 28551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:00:46.580435 28551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.66
I20260927 04:00:46.582924 28551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:46163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.15.185.84:46557/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab
--principal=kudu/127.15.185.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.185.66
--webserver_port=0
--tserver_master_addrs=127.15.185.126:38903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.66
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:00:46.583345 28551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:00:46.583801 28551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:00:46.590415 28559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:00:46.590570 28558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:00:46.590654 28551 server_base.cc:1034] running on GCE node
W20260927 04:00:46.591022 28561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:00:46.591460 28551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:00:46.591751 28551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:00:46.592918 28551 hybrid_clock.cc:648] HybridClock initialized: now 1790481646592895 us; error 34 us; skew 500 ppm
Sep 27 04:00:46 dist-test-slave-glgz krb5kdc[27857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481646, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:00:46.596375 28551 init.cc:377] Logged in from keytab as kudu/127.15.185.66@KRBTEST.COM (short username kudu)
I20260927 04:00:46.596964 28551 webserver.cc:460] Webserver started at http://127.15.185.66:35263/ using document root <none> and password file <none>
I20260927 04:00:46.597244 28551 fs_manager.cc:359] Metadata directory not provided
I20260927 04:00:46.597295 28551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:00:46.597425 28551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 04:00:46 dist-test-slave-glgz krb5kdc[27857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481646, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.66@KRBTEST.COM for HTTP/127.15.185.84@KRBTEST.COM
I20260927 04:00:46.616104 28551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-1/data/instance:
uuid: "a584af7718a84ad2ae3b7f9d4211f873"
format_stamp: "Formatted at 2026-09-27 04:00:46 on dist-test-slave-glgz"
server_key: "e91c1f9e218f77f14371933f6b86f86b"
server_key_iv: "c9418ab93f6f75a0c9c99236db5dcc5c"
server_key_version: "kuduclusterkey@0"
I20260927 04:00:46.616565 28551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-1/wal/instance:
uuid: "a584af7718a84ad2ae3b7f9d4211f873"
format_stamp: "Formatted at 2026-09-27 04:00:46 on dist-test-slave-glgz"
server_key: "e91c1f9e218f77f14371933f6b86f86b"
server_key_iv: "c9418ab93f6f75a0c9c99236db5dcc5c"
server_key_version: "kuduclusterkey@0"
I20260927 04:00:46.619259 28551 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 04:00:46.638058 28568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:00:46.638341 28551 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260927 04:00:46.638447 28551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-1/wal
uuid: "a584af7718a84ad2ae3b7f9d4211f873"
format_stamp: "Formatted at 2026-09-27 04:00:46 on dist-test-slave-glgz"
server_key: "e91c1f9e218f77f14371933f6b86f86b"
server_key_iv: "c9418ab93f6f75a0c9c99236db5dcc5c"
server_key_version: "kuduclusterkey@0"
I20260927 04:00:46.638546 28551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:00:46.661089 28551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:00:46.663610 28551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:00:46.663839 28551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:00:46.664127 28551 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:00:46.664539 28551 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:00:46.664626 28551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:00:46.664721 28551 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:00:46.664783 28551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:00:46.673592 28551 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.66:36119
I20260927 04:00:46.674180 28551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
I20260927 04:00:46.674746 28682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.66:36119 every 8 connection(s)
Sep 27 04:00:46 dist-test-slave-glgz krb5kdc[27857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481646, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.66@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 04:00:46.683506 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 28551
I20260927 04:00:46.683615 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-1/wal/instance
I20260927 04:00:46.683851 16101 external_mini_cluster.cc:1428] Setting key c33635b40ba55ddb695bb91541acd241
I20260927 04:00:46.689148 28548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:46.675656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:57297 (local address 127.15.185.126:38903)
0927 04:00:46.675717 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 04:00:46.675719 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:00:46.676002 (+ 283us) server_negotiation.cc:407] Connection header received
0927 04:00:46.676755 (+ 753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:00:46.676757 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:00:46.676776 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:00:46.676808 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:00:46.677656 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:46.678481 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:46.679954 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:46.680261 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:46.680309 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:46.683215 (+ 2906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:00:46.683217 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:00:46.683218 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:00:46.683226 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:00:46.685627 (+ 2401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:46.687989 (+ 2362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:46.687991 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:46.687991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:46.688057 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:46.688348 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:46.688349 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:46.688350 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:46.688494 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:00:46.688547 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 04:00:46.688943 (+ 396us) server_negotiation.cc:299] Negotiation successful
0927 04:00:46.689044 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 04:00:46.690186 28685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:46.675666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:38903 (local address 127.15.185.66:57297)
0927 04:00:46.675866 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:46.675910 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 04:00:46.676599 (+ 689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:00:46.677004 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:00:46.677008 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:00:46.677266 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:00:46.677543 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:46.677549 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:46.678601 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:46.678602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:46.679848 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:46.679851 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:46.680414 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:46.680415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:46.680556 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:46.681222 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:00:46.681234 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:00:46.683096 (+ 1862us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:00:46.687486 (+ 4390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:00:46.687499 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:00:46.687504 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:00:46.687890 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:00:46.688127 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:00:46.688127 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:00:46.688128 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:00:46.688278 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:00:46.688581 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:00:46.688587 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:00:46.688784 (+ 197us) client_negotiation.cc:769] Sending connection context
0927 04:00:46.689740 (+ 956us) client_negotiation.cc:240] Negotiation successful
0927 04:00:46.689935 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":115,"threads_started":1}
I20260927 04:00:46.690868 28683 heartbeater.cc:347] Connected to a master server at 127.15.185.126:38903
I20260927 04:00:46.690964 28683 heartbeater.cc:464] Registering TS with master...
I20260927 04:00:46.691186 28683 heartbeater.cc:511] Master 127.15.185.126:38903 requested a full tablet report, sending...
I20260927 04:00:46.691622 28333 ts_manager.cc:194] Registered new tserver with Master: a584af7718a84ad2ae3b7f9d4211f873 (127.15.185.66:36119)
I20260927 04:00:46.692149 28333 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.185.66@KRBTEST.COM'} at 127.15.185.66:57297
WARNING: no policy specified for kudu/127.15.185.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab.
Entry for principal kudu/127.15.185.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab.
I20260927 04:00:46.716316 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.185.84:46557/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.67:0
--local_ip_for_outbound_sockets=127.15.185.67
--webserver_interface=127.15.185.67
--webserver_port=0
--tserver_master_addrs=127.15.185.126:38903
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab
--principal=kudu/127.15.185.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:46163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:00:46.839630 28688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:00:46.839998 28688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:00:46.840098 28688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:00:46.841996 28688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:00:46.842135 28688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:00:46.842238 28688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.67
I20260927 04:00:46.844766 28688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:46163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.15.185.84:46557/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab
--principal=kudu/127.15.185.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.185.67
--webserver_port=0
--tserver_master_addrs=127.15.185.126:38903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.67
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:00:46.845269 28688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:00:46.845597 28688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:00:46.852583 28693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:00:46.852583 28694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:00:46.852977 28688 server_base.cc:1034] running on GCE node
W20260927 04:00:46.854095 28696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:00:46.854399 28688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:00:46.854651 28688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:00:46.855813 28688 hybrid_clock.cc:648] HybridClock initialized: now 1790481646855792 us; error 33 us; skew 500 ppm
Sep 27 04:00:46 dist-test-slave-glgz krb5kdc[27857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481646, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:00:46.858944 28688 init.cc:377] Logged in from keytab as kudu/127.15.185.67@KRBTEST.COM (short username kudu)
I20260927 04:00:46.859483 28688 webserver.cc:460] Webserver started at http://127.15.185.67:32873/ using document root <none> and password file <none>
I20260927 04:00:46.859730 28688 fs_manager.cc:359] Metadata directory not provided
I20260927 04:00:46.859794 28688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:00:46.859926 28688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 04:00:46 dist-test-slave-glgz krb5kdc[27857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481646, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.67@KRBTEST.COM for HTTP/127.15.185.84@KRBTEST.COM
I20260927 04:00:46.873843 28688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-2/data/instance:
uuid: "fe8846f2a6a74327bd039989735bb923"
format_stamp: "Formatted at 2026-09-27 04:00:46 on dist-test-slave-glgz"
server_key: "f7100802617a441c14e46099f6db4382"
server_key_iv: "c3bd41c7c6d76c2051cb16a0071bcecd"
server_key_version: "kuduclusterkey@0"
I20260927 04:00:46.874239 28688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-2/wal/instance:
uuid: "fe8846f2a6a74327bd039989735bb923"
format_stamp: "Formatted at 2026-09-27 04:00:46 on dist-test-slave-glgz"
server_key: "f7100802617a441c14e46099f6db4382"
server_key_iv: "c3bd41c7c6d76c2051cb16a0071bcecd"
server_key_version: "kuduclusterkey@0"
I20260927 04:00:46.877044 28688 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 04:00:46.893915 28703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:00:46.899299 28688 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.001s sys 0.000s
I20260927 04:00:46.899459 28688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-2/wal
uuid: "fe8846f2a6a74327bd039989735bb923"
format_stamp: "Formatted at 2026-09-27 04:00:46 on dist-test-slave-glgz"
server_key: "f7100802617a441c14e46099f6db4382"
server_key_iv: "c3bd41c7c6d76c2051cb16a0071bcecd"
server_key_version: "kuduclusterkey@0"
I20260927 04:00:46.899575 28688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:00:46.935447 28688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:00:46.937922 28688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:00:46.938067 28688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:00:46.938378 28688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:00:46.938781 28688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:00:46.938827 28688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:00:46.938862 28688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:00:46.938886 28688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:00:46.947813 28688 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.67:42545
I20260927 04:00:46.948243 28688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
I20260927 04:00:46.954211 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 28688
I20260927 04:00:46.954341 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-2/wal/instance
I20260927 04:00:46.954546 16101 external_mini_cluster.cc:1428] Setting key dd3a22284b506e363ece4ab3dcf169a8
I20260927 04:00:46.956934 28816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.67:42545 every 8 connection(s)
Sep 27 04:00:46 dist-test-slave-glgz krb5kdc[27857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481646, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.67@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 04:00:46.969285 28548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:46.958317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:51281 (local address 127.15.185.126:38903)
0927 04:00:46.958393 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0927 04:00:46.958395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:00:46.958679 (+ 284us) server_negotiation.cc:407] Connection header received
0927 04:00:46.959677 (+ 998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:00:46.959679 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:00:46.959712 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:00:46.959744 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:00:46.960718 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:46.961525 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:46.962929 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:46.963300 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:46.963352 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:46.965986 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:00:46.965988 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:00:46.965989 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:00:46.965997 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:00:46.967698 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:46.968176 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:46.968177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:46.968177 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:46.968233 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:46.968509 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:46.968510 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:46.968510 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:46.968638 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:00:46.968676 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0927 04:00:46.969070 (+ 394us) server_negotiation.cc:299] Negotiation successful
0927 04:00:46.969183 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 04:00:46.969473 28819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:46.958280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:38903 (local address 127.15.185.67:51281)
0927 04:00:46.958499 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:46.958578 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0927 04:00:46.959499 (+ 921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:00:46.960022 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:00:46.960027 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:00:46.960306 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:00:46.960603 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:46.960610 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:46.961645 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:46.961647 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:46.962826 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:46.962829 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:46.963461 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:46.963462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:46.963607 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:46.964226 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:00:46.964237 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:00:46.965873 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:00:46.967799 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:00:46.967806 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:00:46.967809 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:00:46.968094 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:00:46.968314 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:00:46.968315 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:00:46.968315 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:00:46.968443 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:00:46.968719 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:00:46.968724 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:00:46.968888 (+ 164us) client_negotiation.cc:769] Sending connection context
0927 04:00:46.969006 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 04:00:46.969161 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":102,"threads_started":1}
I20260927 04:00:46.969887 28817 heartbeater.cc:347] Connected to a master server at 127.15.185.126:38903
I20260927 04:00:46.969980 28817 heartbeater.cc:464] Registering TS with master...
I20260927 04:00:46.970165 28817 heartbeater.cc:511] Master 127.15.185.126:38903 requested a full tablet report, sending...
I20260927 04:00:46.970504 28333 ts_manager.cc:194] Registered new tserver with Master: fe8846f2a6a74327bd039989735bb923 (127.15.185.67:42545)
I20260927 04:00:46.970957 28333 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.185.67@KRBTEST.COM'} at 127.15.185.67:51281
I20260927 04:00:46.977003 16101 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 04:00:46.977890 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 28414
I20260927 04:00:46.984228 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 28551
I20260927 04:00:46.990775 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 28688
I20260927 04:00:46.997041 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 28301
I20260927 04:00:47.004230 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:38903
--webserver_interface=127.15.185.126
--webserver_port=46047
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:46163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.185.126:38903
--ranger_config_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.185.84:46557/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.15.185.84:46557/kms with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:00:47.116276 28820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:00:47.116518 28820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:00:47.116541 28820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:00:47.117807 28820 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:00:47.117865 28820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:00:47.117880 28820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:00:47.117893 28820 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:00:47.117905 28820 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:00:47.119748 28820 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:46163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.15.185.84:46557/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.15.185.126:38903
--ranger_config_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:38903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.185.126
--webserver_port=46047
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:00:47.120033 28820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:00:47.120272 28820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:00:47.126163 28826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:00:47.126157 28825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:00:47.126163 28828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:00:47.126587 28820 server_base.cc:1034] running on GCE node
I20260927 04:00:47.126884 28820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:00:47.127236 28820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:00:47.128414 28820 hybrid_clock.cc:648] HybridClock initialized: now 1790481647128400 us; error 31 us; skew 500 ppm
Sep 27 04:00:47 dist-test-slave-glgz krb5kdc[27857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481647, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:00:47.131127 28820 init.cc:377] Logged in from keytab as kudu/127.15.185.126@KRBTEST.COM (short username kudu)
I20260927 04:00:47.131577 28820 webserver.cc:460] Webserver started at http://127.15.185.126:46047/ using document root <none> and password file <none>
I20260927 04:00:47.131817 28820 fs_manager.cc:359] Metadata directory not provided
I20260927 04:00:47.131889 28820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 04:00:47 dist-test-slave-glgz krb5kdc[27857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481647, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.126@KRBTEST.COM for HTTP/127.15.185.84@KRBTEST.COM
I20260927 04:00:47.194063 28820 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:00:47.194937 28836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:00:47.195225 28820 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:00:47.195336 28820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/master-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/master-0/wal
uuid: "aff4fb0709a14bfbbc2e9060c6d43b72"
format_stamp: "Formatted at 2026-09-27 04:00:45 on dist-test-slave-glgz"
server_key: "0f43b0da6edd7820ba2a4e5b585a6c95"
server_key_iv: "1ea03daf82bde83bc6945a013afcf56f"
server_key_version: "kuduclusterkey@0"
I20260927 04:00:47.195811 28820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:00:47.210983 28820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:00:47.213464 28820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:00:47.213636 28820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:00:47.218791 28820 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.126:38903
I20260927 04:00:47.218904 28888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.126:38903 every 8 connection(s)
I20260927 04:00:47.219372 28820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
I20260927 04:00:47.220664 28889 sys_catalog.cc:263] Verifying existing consensus state
I20260927 04:00:47.221257 28889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aff4fb0709a14bfbbc2e9060c6d43b72: Bootstrap starting.
I20260927 04:00:47.221410 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 28820
I20260927 04:00:47.221686 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.185.84:46557/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.65:36243
--local_ip_for_outbound_sockets=127.15.185.65
--tserver_master_addrs=127.15.185.126:38903
--webserver_port=36205
--webserver_interface=127.15.185.65
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab
--principal=kudu/127.15.185.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:46163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.15.185.84:46557/kms with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 04:00:47.229406 28889 log.cc:824] T 00000000000000000000000000000000 P aff4fb0709a14bfbbc2e9060c6d43b72: Log is configured to *not* fsync() on all Append() calls
I20260927 04:00:47.231384 28889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aff4fb0709a14bfbbc2e9060c6d43b72: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 04:00:47.231693 28889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aff4fb0709a14bfbbc2e9060c6d43b72: Bootstrap complete.
I20260927 04:00:47.234143 28889 raft_consensus.cc:348] T 00000000000000000000000000000000 P aff4fb0709a14bfbbc2e9060c6d43b72 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff4fb0709a14bfbbc2e9060c6d43b72" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 38903 } }
I20260927 04:00:47.234427 28889 raft_consensus.cc:729] T 00000000000000000000000000000000 P aff4fb0709a14bfbbc2e9060c6d43b72 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aff4fb0709a14bfbbc2e9060c6d43b72, State: Initialized, Role: FOLLOWER
I20260927 04:00:47.234611 28889 consensus_queue.cc:260] T 00000000000000000000000000000000 P aff4fb0709a14bfbbc2e9060c6d43b72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff4fb0709a14bfbbc2e9060c6d43b72" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 38903 } }
I20260927 04:00:47.234680 28889 raft_consensus.cc:388] T 00000000000000000000000000000000 P aff4fb0709a14bfbbc2e9060c6d43b72 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:00:47.234735 28889 raft_consensus.cc:482] T 00000000000000000000000000000000 P aff4fb0709a14bfbbc2e9060c6d43b72 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:00:47.234784 28889 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aff4fb0709a14bfbbc2e9060c6d43b72 [term 1 FOLLOWER]: Advancing to term 2
I20260927 04:00:47.235934 28889 raft_consensus.cc:504] T 00000000000000000000000000000000 P aff4fb0709a14bfbbc2e9060c6d43b72 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff4fb0709a14bfbbc2e9060c6d43b72" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 38903 } }
I20260927 04:00:47.236027 28889 leader_election.cc:302] T 00000000000000000000000000000000 P aff4fb0709a14bfbbc2e9060c6d43b72 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aff4fb0709a14bfbbc2e9060c6d43b72; no voters:
I20260927 04:00:47.236233 28889 leader_election.cc:288] T 00000000000000000000000000000000 P aff4fb0709a14bfbbc2e9060c6d43b72 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 04:00:47.236269 28893 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aff4fb0709a14bfbbc2e9060c6d43b72 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 04:00:47.236384 28893 raft_consensus.cc:686] T 00000000000000000000000000000000 P aff4fb0709a14bfbbc2e9060c6d43b72 [term 2 LEADER]: Becoming Leader. State: Replica: aff4fb0709a14bfbbc2e9060c6d43b72, State: Running, Role: LEADER
I20260927 04:00:47.236480 28893 consensus_queue.cc:237] T 00000000000000000000000000000000 P aff4fb0709a14bfbbc2e9060c6d43b72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff4fb0709a14bfbbc2e9060c6d43b72" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 38903 } }
I20260927 04:00:47.236515 28889 sys_catalog.cc:564] T 00000000000000000000000000000000 P aff4fb0709a14bfbbc2e9060c6d43b72 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:00:47.236732 28894 sys_catalog.cc:455] T 00000000000000000000000000000000 P aff4fb0709a14bfbbc2e9060c6d43b72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "aff4fb0709a14bfbbc2e9060c6d43b72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff4fb0709a14bfbbc2e9060c6d43b72" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 38903 } } }
I20260927 04:00:47.236824 28894 sys_catalog.cc:458] T 00000000000000000000000000000000 P aff4fb0709a14bfbbc2e9060c6d43b72 [sys.catalog]: This master's current role is: LEADER
I20260927 04:00:47.237043 28895 sys_catalog.cc:455] T 00000000000000000000000000000000 P aff4fb0709a14bfbbc2e9060c6d43b72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aff4fb0709a14bfbbc2e9060c6d43b72. Latest consensus state: current_term: 2 leader_uuid: "aff4fb0709a14bfbbc2e9060c6d43b72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aff4fb0709a14bfbbc2e9060c6d43b72" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 38903 } } }
I20260927 04:00:47.237138 28895 sys_catalog.cc:458] T 00000000000000000000000000000000 P aff4fb0709a14bfbbc2e9060c6d43b72 [sys.catalog]: This master's current role is: LEADER
I20260927 04:00:47.237277 28897 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:00:47.237757 28897 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:00:47.237888 28897 catalog_manager.cc:1259] Loaded cluster ID: a7b4ab978a7643e9967a6f35bf499c48
I20260927 04:00:47.237924 28897 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:00:47.238104 28889 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 04:00:47.239176 28897 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:00:47.241267 28897 catalog_manager.cc:5879] T 00000000000000000000000000000000 P aff4fb0709a14bfbbc2e9060c6d43b72: Loaded TSK: 0
I20260927 04:00:47.241779 28897 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 04:00:47.334713 28891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:00:47.335032 28891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:00:47.335129 28891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:00:47.336975 28891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:00:47.337044 28891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:00:47.337088 28891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.65
I20260927 04:00:47.339444 28891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:46163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.15.185.84:46557/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab
--principal=kudu/127.15.185.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.65:36243
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.185.65
--webserver_port=36205
--tserver_master_addrs=127.15.185.126:38903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.65
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:00:47.339694 28891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:00:47.339989 28891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:00:47.348482 28929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:00:47.348869 28891 server_base.cc:1034] running on GCE node
W20260927 04:00:47.349431 28927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:00:47.349679 28926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:00:47.349996 28891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:00:47.350296 28891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:00:47.351466 28891 hybrid_clock.cc:648] HybridClock initialized: now 1790481647351442 us; error 36 us; skew 500 ppm
Sep 27 04:00:47 dist-test-slave-glgz krb5kdc[27857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481647, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:00:47.354858 28891 init.cc:377] Logged in from keytab as kudu/127.15.185.65@KRBTEST.COM (short username kudu)
I20260927 04:00:47.355446 28891 webserver.cc:460] Webserver started at http://127.15.185.65:36205/ using document root <none> and password file <none>
I20260927 04:00:47.355717 28891 fs_manager.cc:359] Metadata directory not provided
I20260927 04:00:47.355804 28891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 04:00:47 dist-test-slave-glgz krb5kdc[27857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481647, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.65@KRBTEST.COM for HTTP/127.15.185.84@KRBTEST.COM
I20260927 04:00:47.390331 28891 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.000s sys 0.001s
I20260927 04:00:47.391525 28937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:00:47.391775 28891 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:00:47.391867 28891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-0/wal
uuid: "9069f7895034488b852820a86b6ac57c"
format_stamp: "Formatted at 2026-09-27 04:00:46 on dist-test-slave-glgz"
server_key: "977779661d056c4e8e055e954a0e1c15"
server_key_iv: "afadfee72a603d90cc6e89b0e2da8d72"
server_key_version: "kuduclusterkey@0"
I20260927 04:00:47.392379 28891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:00:47.404826 28891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:00:47.408694 28891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:00:47.408944 28891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:00:47.409299 28891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:00:47.409724 28891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:00:47.409816 28891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:00:47.409925 28891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:00:47.409997 28891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:00:47.417001 28891 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.65:36243
I20260927 04:00:47.417521 28891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
I20260927 04:00:47.417829 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 28891
I20260927 04:00:47.418164 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.185.84:46557/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.66:36119
--local_ip_for_outbound_sockets=127.15.185.66
--tserver_master_addrs=127.15.185.126:38903
--webserver_port=35263
--webserver_interface=127.15.185.66
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab
--principal=kudu/127.15.185.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:46163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.15.185.84:46557/kms with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 04:00:47.419514 29050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.65:36243 every 8 connection(s)
Sep 27 04:00:47 dist-test-slave-glgz krb5kdc[27857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481647, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.65@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 04:00:47.434456 29054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:47.420400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:38903 (local address 127.15.185.65:48021)
0927 04:00:47.420908 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:47.420951 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 04:00:47.421540 (+ 589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:00:47.423105 (+ 1565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:00:47.423111 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:00:47.423339 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:00:47.423641 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:47.423647 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:47.425386 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:47.425387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:47.426612 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:47.426615 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:47.427594 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:47.427595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:47.427791 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:47.428487 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:00:47.428497 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:00:47.430169 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:00:47.432666 (+ 2497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:00:47.432672 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:00:47.432674 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:00:47.432974 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:00:47.433241 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:00:47.433241 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:00:47.433242 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:00:47.433355 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:00:47.433814 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:00:47.433818 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:00:47.433958 (+ 140us) client_negotiation.cc:769] Sending connection context
0927 04:00:47.434055 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 04:00:47.434215 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":91,"threads_started":1}
I20260927 04:00:47.435462 29055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:47.420520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:48021 (local address 127.15.185.126:38903)
0927 04:00:47.422530 (+ 2010us) server_negotiation.cc:206] Beginning negotiation
0927 04:00:47.422541 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 04:00:47.422586 (+ 45us) server_negotiation.cc:407] Connection header received
0927 04:00:47.422754 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:00:47.422762 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:00:47.422827 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:00:47.423008 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:00:47.423782 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:47.425248 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:47.426723 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:47.427469 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:47.427603 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:47.430329 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:00:47.430333 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:00:47.430334 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:00:47.430346 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:00:47.432573 (+ 2227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:47.433060 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:47.433063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:47.433064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:47.433163 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:47.433417 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:47.433417 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:47.433417 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:47.433713 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:00:47.433788 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 04:00:47.435134 (+ 1346us) server_negotiation.cc:299] Negotiation successful
0927 04:00:47.435265 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1944,"thread_start_us":105,"threads_started":1}
I20260927 04:00:47.436414 29051 heartbeater.cc:347] Connected to a master server at 127.15.185.126:38903
I20260927 04:00:47.436507 29051 heartbeater.cc:464] Registering TS with master...
I20260927 04:00:47.436681 29051 heartbeater.cc:511] Master 127.15.185.126:38903 requested a full tablet report, sending...
W20260927 04:00:47.437502 29051 heartbeater.cc:644] Failed to heartbeat to 127.15.185.126:38903 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 04:00:47.437562 29051 heartbeater.cc:464] Registering TS with master...
I20260927 04:00:47.437615 29051 heartbeater.cc:511] Master 127.15.185.126:38903 requested a full tablet report, sending...
I20260927 04:00:47.437964 29051 heartbeater.cc:464] Registering TS with master...
I20260927 04:00:47.438028 29051 heartbeater.cc:511] Master 127.15.185.126:38903 requested a full tablet report, sending...
W20260927 04:00:47.438319 29051 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 04:00:47.543092 29053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:00:47.543373 29053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:00:47.543416 29053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:00:47.544705 29053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:00:47.544775 29053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:00:47.544827 29053 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.66
I20260927 04:00:47.546561 29053 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:46163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.15.185.84:46557/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab
--principal=kudu/127.15.185.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.66:36119
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.185.66
--webserver_port=35263
--tserver_master_addrs=127.15.185.126:38903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29053
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.66
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:00:47.546835 29053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:00:47.547112 29053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:00:47.553320 29060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:00:47.553511 29053 server_base.cc:1034] running on GCE node
W20260927 04:00:47.553316 29063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:00:47.553426 29061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:00:47.554049 29053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:00:47.554322 29053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:00:47.555487 29053 hybrid_clock.cc:648] HybridClock initialized: now 1790481647555473 us; error 31 us; skew 500 ppm
Sep 27 04:00:47 dist-test-slave-glgz krb5kdc[27857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481647, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:00:47.558604 29053 init.cc:377] Logged in from keytab as kudu/127.15.185.66@KRBTEST.COM (short username kudu)
I20260927 04:00:47.559207 29053 webserver.cc:460] Webserver started at http://127.15.185.66:35263/ using document root <none> and password file <none>
I20260927 04:00:47.559508 29053 fs_manager.cc:359] Metadata directory not provided
I20260927 04:00:47.559598 29053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 04:00:47 dist-test-slave-glgz krb5kdc[27857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481647, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.66@KRBTEST.COM for HTTP/127.15.185.84@KRBTEST.COM
I20260927 04:00:47.579376 29053 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260927 04:00:47.580299 29071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:00:47.580544 29053 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:00:47.580667 29053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-1/wal
uuid: "a584af7718a84ad2ae3b7f9d4211f873"
format_stamp: "Formatted at 2026-09-27 04:00:46 on dist-test-slave-glgz"
server_key: "e91c1f9e218f77f14371933f6b86f86b"
server_key_iv: "c9418ab93f6f75a0c9c99236db5dcc5c"
server_key_version: "kuduclusterkey@0"
I20260927 04:00:47.581193 29053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:00:47.597645 29053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:00:47.600198 29053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:00:47.600378 29053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:00:47.600726 29053 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:00:47.601137 29053 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:00:47.601186 29053 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:00:47.601218 29053 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:00:47.601287 29053 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:00:47.608860 29053 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.66:36119
I20260927 04:00:47.608897 29184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.66:36119 every 8 connection(s)
I20260927 04:00:47.609372 29053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
Sep 27 04:00:47 dist-test-slave-glgz krb5kdc[27857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481647, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.66@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 04:00:47.617197 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 29053
I20260927 04:00:47.617475 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.185.84:46557/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.67:42545
--local_ip_for_outbound_sockets=127.15.185.67
--tserver_master_addrs=127.15.185.126:38903
--webserver_port=32873
--webserver_interface=127.15.185.67
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab
--principal=kudu/127.15.185.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:46163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.15.185.84:46557/kms with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 04:00:47.620811 29055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:47.610147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:36889 (local address 127.15.185.126:38903)
0927 04:00:47.610217 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 04:00:47.610220 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:00:47.610452 (+ 232us) server_negotiation.cc:407] Connection header received
0927 04:00:47.610916 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:00:47.610918 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:00:47.610934 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:00:47.611025 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:00:47.611629 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:47.612428 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:47.613462 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:47.613695 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:47.613778 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:47.616155 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:00:47.616156 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:00:47.616156 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:00:47.616164 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:00:47.619289 (+ 3125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:47.619763 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:47.619765 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:47.619765 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:47.619825 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:47.620089 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:47.620090 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:47.620090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:47.620212 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:00:47.620285 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 04:00:47.620619 (+ 334us) server_negotiation.cc:299] Negotiation successful
0927 04:00:47.620714 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 04:00:47.621642 29187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:47.610132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:38903 (local address 127.15.185.66:36889)
0927 04:00:47.610293 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:47.610335 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 04:00:47.610806 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:00:47.611134 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:00:47.611139 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:00:47.611323 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:00:47.611530 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:47.611534 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:47.612526 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:47.612528 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:47.613379 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:47.613382 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:47.613807 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:47.613809 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:47.613931 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:47.614467 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:00:47.614476 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:00:47.616045 (+ 1569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:00:47.619387 (+ 3342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:00:47.619395 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:00:47.619398 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:00:47.619684 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:00:47.619899 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:00:47.619900 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:00:47.619901 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:00:47.620023 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:00:47.620276 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:00:47.620282 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:00:47.620452 (+ 170us) client_negotiation.cc:769] Sending connection context
0927 04:00:47.621215 (+ 763us) client_negotiation.cc:240] Negotiation successful
0927 04:00:47.621399 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":92,"threads_started":1}
I20260927 04:00:47.622349 29185 heartbeater.cc:347] Connected to a master server at 127.15.185.126:38903
I20260927 04:00:47.622470 29185 heartbeater.cc:464] Registering TS with master...
I20260927 04:00:47.622654 29185 heartbeater.cc:511] Master 127.15.185.126:38903 requested a full tablet report, sending...
W20260927 04:00:47.623324 29185 heartbeater.cc:644] Failed to heartbeat to 127.15.185.126:38903 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 04:00:47.623401 29185 heartbeater.cc:464] Registering TS with master...
I20260927 04:00:47.623476 29185 heartbeater.cc:511] Master 127.15.185.126:38903 requested a full tablet report, sending...
I20260927 04:00:47.623752 29185 heartbeater.cc:464] Registering TS with master...
I20260927 04:00:47.623824 29185 heartbeater.cc:511] Master 127.15.185.126:38903 requested a full tablet report, sending...
W20260927 04:00:47.624126 29185 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 04:00:47.741456 29188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:00:47.741770 29188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:00:47.741817 29188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:00:47.743788 29188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:00:47.743856 29188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:00:47.743906 29188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.67
I20260927 04:00:47.746162 29188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:46163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.15.185.84:46557/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab
--principal=kudu/127.15.185.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.67:42545
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.185.67
--webserver_port=32873
--tserver_master_addrs=127.15.185.126:38903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.67
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:00:47.746444 29188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:00:47.746727 29188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:00:47.752409 29195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:00:47.752415 29197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:00:47.752619 29188 server_base.cc:1034] running on GCE node
W20260927 04:00:47.752547 29194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:00:47.752868 29188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:00:47.753165 29188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:00:47.754367 29188 hybrid_clock.cc:648] HybridClock initialized: now 1790481647754359 us; error 33 us; skew 500 ppm
Sep 27 04:00:47 dist-test-slave-glgz krb5kdc[27857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481647, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:00:47.758769 29188 init.cc:377] Logged in from keytab as kudu/127.15.185.67@KRBTEST.COM (short username kudu)
I20260927 04:00:47.759353 29188 webserver.cc:460] Webserver started at http://127.15.185.67:32873/ using document root <none> and password file <none>
I20260927 04:00:47.759657 29188 fs_manager.cc:359] Metadata directory not provided
I20260927 04:00:47.759740 29188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 04:00:47 dist-test-slave-glgz krb5kdc[27857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481647, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.67@KRBTEST.COM for HTTP/127.15.185.84@KRBTEST.COM
I20260927 04:00:47.780718 29188 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.001s
I20260927 04:00:47.781656 29205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:00:47.781883 29188 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:00:47.782251 29188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-2/wal
uuid: "fe8846f2a6a74327bd039989735bb923"
format_stamp: "Formatted at 2026-09-27 04:00:46 on dist-test-slave-glgz"
server_key: "f7100802617a441c14e46099f6db4382"
server_key_iv: "c3bd41c7c6d76c2051cb16a0071bcecd"
server_key_version: "kuduclusterkey@0"
I20260927 04:00:47.782611 29188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:00:47.819842 29188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:00:47.822340 29188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:00:47.822512 29188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:00:47.822763 29188 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:00:47.823132 29188 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:00:47.823181 29188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:00:47.823216 29188 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:00:47.823235 29188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:00:47.832823 29188 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.67:42545
I20260927 04:00:47.832908 29318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.67:42545 every 8 connection(s)
I20260927 04:00:47.833242 29188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
I20260927 04:00:47.834214 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 29188
Sep 27 04:00:47 dist-test-slave-glgz krb5kdc[27857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481647, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.67@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 04:00:47.846909 29055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:47.834065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:47609 (local address 127.15.185.126:38903)
0927 04:00:47.834125 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 04:00:47.834128 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:00:47.834490 (+ 362us) server_negotiation.cc:407] Connection header received
0927 04:00:47.835107 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:00:47.835109 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:00:47.835121 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:00:47.835182 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:00:47.835961 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:47.836763 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:47.838282 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:47.838559 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:47.838620 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:47.844172 (+ 5552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:00:47.844173 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:00:47.844174 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:00:47.844181 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:00:47.845575 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:47.845993 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:47.845994 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:47.845995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:47.846045 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:47.846294 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:47.846295 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:47.846295 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:47.846402 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:00:47.846470 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 04:00:47.846737 (+ 267us) server_negotiation.cc:299] Negotiation successful
0927 04:00:47.846814 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 04:00:47.847157 29321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:47.834082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:38903 (local address 127.15.185.67:47609)
0927 04:00:47.834254 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:47.834379 (+ 125us) client_negotiation.cc:174] Beginning negotiation
0927 04:00:47.834973 (+ 594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:00:47.835306 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:00:47.835311 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:00:47.835561 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:00:47.835851 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:47.835859 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:47.836853 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:47.836855 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:47.838185 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:47.838188 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:47.838698 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:47.838699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:47.838835 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:47.839390 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:00:47.839400 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:00:47.844077 (+ 4677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:00:47.845673 (+ 1596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:00:47.845680 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:00:47.845682 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:00:47.845909 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:00:47.846112 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:00:47.846113 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:00:47.846113 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:00:47.846230 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:00:47.846464 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:00:47.846470 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:00:47.846612 (+ 142us) client_negotiation.cc:769] Sending connection context
0927 04:00:47.846719 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 04:00:47.846879 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":93,"threads_started":1}
I20260927 04:00:47.847590 29319 heartbeater.cc:347] Connected to a master server at 127.15.185.126:38903
I20260927 04:00:47.847692 29319 heartbeater.cc:464] Registering TS with master...
I20260927 04:00:47.847895 29319 heartbeater.cc:511] Master 127.15.185.126:38903 requested a full tablet report, sending...
W20260927 04:00:47.848474 29319 heartbeater.cc:644] Failed to heartbeat to 127.15.185.126:38903 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 04:00:47.848548 29319 heartbeater.cc:464] Registering TS with master...
I20260927 04:00:47.848610 29319 heartbeater.cc:511] Master 127.15.185.126:38903 requested a full tablet report, sending...
I20260927 04:00:47.848898 29319 heartbeater.cc:464] Registering TS with master...
I20260927 04:00:47.848973 29319 heartbeater.cc:511] Master 127.15.185.126:38903 requested a full tablet report, sending...
W20260927 04:00:47.849254 29319 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 04:00:47.850162 29323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:47.834636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52636 (local address 127.15.185.126:38903)
0927 04:00:47.835354 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0927 04:00:47.835355 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 04:00:47.835361 (+ 6us) server_negotiation.cc:407] Connection header received
0927 04:00:47.835378 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:00:47.835378 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:00:47.835385 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:00:47.835459 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:00:47.836092 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:47.836844 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:47.837874 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:47.838092 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:47.838174 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:47.840026 (+ 1852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:00:47.840027 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:00:47.840028 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:00:47.840039 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:00:47.841735 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:47.842794 (+ 1059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:47.842795 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:47.842796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:47.842871 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:47.844192 (+ 1321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:47.844193 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:47.844193 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:47.844363 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:00:47.844520 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 04:00:47.849968 (+ 5448us) server_negotiation.cc:299] Negotiation successful
0927 04:00:47.850058 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":704,"thread_start_us":70,"threads_started":1}
Sep 27 04:00:48 dist-test-slave-glgz krb5kdc[27857](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790481648, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:00:48.438496 29051 heartbeater.cc:464] Registering TS with master...
I20260927 04:00:48.438650 29051 heartbeater.cc:511] Master 127.15.185.126:38903 requested a full tablet report, sending...
I20260927 04:00:48.439666 28853 ts_manager.cc:194] Registered new tserver with Master: 9069f7895034488b852820a86b6ac57c (127.15.185.65:36243)
I20260927 04:00:48.440698 28853 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.185.65@KRBTEST.COM'} at 127.15.185.65:48021
I20260927 04:00:48.624334 29185 heartbeater.cc:464] Registering TS with master...
I20260927 04:00:48.624511 29185 heartbeater.cc:511] Master 127.15.185.126:38903 requested a full tablet report, sending...
I20260927 04:00:48.625038 28853 ts_manager.cc:194] Registered new tserver with Master: a584af7718a84ad2ae3b7f9d4211f873 (127.15.185.66:36119)
I20260927 04:00:48.625655 28853 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.185.66@KRBTEST.COM'} at 127.15.185.66:36889
Sep 27 04:00:48 dist-test-slave-glgz krb5kdc[27857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481648, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.126@KRBTEST.COM for HTTP/127.15.185.84@KRBTEST.COM
I20260927 04:00:48.849439 29319 heartbeater.cc:464] Registering TS with master...
I20260927 04:00:48.849609 29319 heartbeater.cc:511] Master 127.15.185.126:38903 requested a full tablet report, sending...
I20260927 04:00:48.849968 28853 ts_manager.cc:194] Registered new tserver with Master: fe8846f2a6a74327bd039989735bb923 (127.15.185.67:42545)
I20260927 04:00:48.850370 28853 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.185.67@KRBTEST.COM'} at 127.15.185.67:47609
I20260927 04:00:48.853542 16101 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 04:00:48.861035 29341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:48.854508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52640 (local address 127.15.185.126:38903)
0927 04:00:48.854617 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 04:00:48.854620 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:00:48.854699 (+ 79us) server_negotiation.cc:407] Connection header received
0927 04:00:48.854757 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:00:48.854758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:00:48.854770 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:00:48.854834 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:00:48.855475 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:48.855985 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:48.856709 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:48.856932 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:48.857017 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:48.858248 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:00:48.858250 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:00:48.858250 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:00:48.858258 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:00:48.859794 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:48.860157 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:48.860158 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:48.860159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:48.860216 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:48.860500 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:48.860501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:48.860502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:48.860605 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:00:48.860716 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 04:00:48.860811 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 04:00:48.860903 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":58,"threads_started":1}
I20260927 04:00:48.862960 28853 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52640:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 04:00:48.864749 28853 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 04:00:48.875908 29347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:48.867598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.66:36119 (local address 127.0.0.1:52654)
0927 04:00:48.871500 (+ 3902us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:48.871510 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 04:00:48.871555 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:00:48.871840 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:00:48.871841 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:00:48.871843 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:00:48.871897 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:48.871898 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:48.873053 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:48.873053 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:48.874406 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:48.874409 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:48.875546 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:48.875547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:48.875651 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:48.875670 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 04:00:48.875737 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 04:00:48.875802 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3888,"thread_start_us":96,"threads_started":1}
I20260927 04:00:48.876263 29349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:48.867985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52654 (local address 127.15.185.66:36119)
0927 04:00:48.868111 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 04:00:48.868116 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:00:48.871620 (+ 3504us) server_negotiation.cc:407] Connection header received
0927 04:00:48.871660 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:00:48.871663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:00:48.871680 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:00:48.871748 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:00:48.871971 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:48.872970 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:48.874512 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:48.875333 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:48.875438 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:48.875511 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 04:00:48.876139 (+ 628us) server_negotiation.cc:299] Negotiation successful
0927 04:00:48.876183 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":70,"threads_started":1}
I20260927 04:00:48.877044 29119 tablet_service.cc:1467] Processing CreateTablet for tablet 87c881a31862415ab0fdfd9bb3c989ee (DEFAULT_TABLE table=test-table [id=43535011f96f44c3bd97bce2af6ccd2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:00:48.877434 29119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87c881a31862415ab0fdfd9bb3c989ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:00:48.880790 29347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:48.867798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.67:42545 (local address 127.0.0.1:51560)
0927 04:00:48.876006 (+ 8208us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:48.876018 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 04:00:48.876064 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:00:48.876884 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:00:48.876885 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:00:48.876886 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:00:48.876916 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:48.876917 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:48.878100 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:48.878101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:48.879487 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:48.879490 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:48.880542 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:48.880544 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:48.880607 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:48.880611 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 04:00:48.880661 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 04:00:48.880706 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8199,"thread_start_us":54,"threads_started":1}
I20260927 04:00:48.881012 29348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:48.867559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.65:36243 (local address 127.0.0.1:37032)
0927 04:00:48.871250 (+ 3691us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:48.871293 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 04:00:48.871382 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:00:48.871706 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:00:48.871710 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:00:48.871712 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:00:48.871792 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:48.871794 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:48.873334 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:48.873334 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:48.874406 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:48.874409 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:48.875881 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:48.875882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:48.875940 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:48.875944 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 04:00:48.880901 (+ 4957us) client_negotiation.cc:240] Negotiation successful
0927 04:00:48.880946 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3663,"thread_start_us":89,"threads_started":1}
I20260927 04:00:48.881417 29351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:48.868165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51560 (local address 127.15.185.67:42545)
0927 04:00:48.868281 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 04:00:48.868284 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:00:48.876261 (+ 7977us) server_negotiation.cc:407] Connection header received
0927 04:00:48.876677 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:00:48.876688 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:00:48.876712 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:00:48.876797 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:00:48.876991 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:48.877999 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:48.879583 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:48.880362 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:48.880448 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:48.880507 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 04:00:48.881274 (+ 767us) server_negotiation.cc:299] Negotiation successful
0927 04:00:48.881333 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":58,"threads_started":1}
I20260927 04:00:48.881718 29350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:48.868003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37032 (local address 127.15.185.65:36243)
0927 04:00:48.871178 (+ 3175us) server_negotiation.cc:206] Beginning negotiation
0927 04:00:48.871182 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:00:48.871361 (+ 179us) server_negotiation.cc:407] Connection header received
0927 04:00:48.871471 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:00:48.871474 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:00:48.871492 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:00:48.871572 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:00:48.871878 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:48.873251 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:48.874512 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:48.875744 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:48.875799 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:48.875851 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 04:00:48.881584 (+ 5733us) server_negotiation.cc:299] Negotiation successful
0927 04:00:48.881638 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3131,"thread_start_us":67,"threads_started":1}
I20260927 04:00:48.882184 29253 tablet_service.cc:1467] Processing CreateTablet for tablet 87c881a31862415ab0fdfd9bb3c989ee (DEFAULT_TABLE table=test-table [id=43535011f96f44c3bd97bce2af6ccd2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:00:48.882553 29253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87c881a31862415ab0fdfd9bb3c989ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:00:48.882601 28985 tablet_service.cc:1467] Processing CreateTablet for tablet 87c881a31862415ab0fdfd9bb3c989ee (DEFAULT_TABLE table=test-table [id=43535011f96f44c3bd97bce2af6ccd2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:00:48.882898 28985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87c881a31862415ab0fdfd9bb3c989ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:00:48.886850 29354 tablet_bootstrap.cc:492] T 87c881a31862415ab0fdfd9bb3c989ee P 9069f7895034488b852820a86b6ac57c: Bootstrap starting.
I20260927 04:00:48.887219 29353 tablet_bootstrap.cc:492] T 87c881a31862415ab0fdfd9bb3c989ee P fe8846f2a6a74327bd039989735bb923: Bootstrap starting.
I20260927 04:00:48.887722 29354 tablet_bootstrap.cc:654] T 87c881a31862415ab0fdfd9bb3c989ee P 9069f7895034488b852820a86b6ac57c: Neither blocks nor log segments found. Creating new log.
I20260927 04:00:48.887900 29353 tablet_bootstrap.cc:654] T 87c881a31862415ab0fdfd9bb3c989ee P fe8846f2a6a74327bd039989735bb923: Neither blocks nor log segments found. Creating new log.
I20260927 04:00:48.888129 29354 log.cc:824] T 87c881a31862415ab0fdfd9bb3c989ee P 9069f7895034488b852820a86b6ac57c: Log is configured to *not* fsync() on all Append() calls
I20260927 04:00:48.888263 29353 log.cc:824] T 87c881a31862415ab0fdfd9bb3c989ee P fe8846f2a6a74327bd039989735bb923: Log is configured to *not* fsync() on all Append() calls
I20260927 04:00:48.888996 29353 tablet_bootstrap.cc:492] T 87c881a31862415ab0fdfd9bb3c989ee P fe8846f2a6a74327bd039989735bb923: No bootstrap required, opened a new log
I20260927 04:00:48.889081 29353 ts_tablet_manager.cc:1397] T 87c881a31862415ab0fdfd9bb3c989ee P fe8846f2a6a74327bd039989735bb923: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 04:00:48.891336 29352 tablet_bootstrap.cc:492] T 87c881a31862415ab0fdfd9bb3c989ee P a584af7718a84ad2ae3b7f9d4211f873: Bootstrap starting.
I20260927 04:00:48.891739 29353 raft_consensus.cc:348] T 87c881a31862415ab0fdfd9bb3c989ee P fe8846f2a6a74327bd039989735bb923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9069f7895034488b852820a86b6ac57c" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 36243 } } peers { permanent_uuid: "a584af7718a84ad2ae3b7f9d4211f873" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 36119 } } peers { permanent_uuid: "fe8846f2a6a74327bd039989735bb923" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 42545 } }
I20260927 04:00:48.891899 29353 raft_consensus.cc:374] T 87c881a31862415ab0fdfd9bb3c989ee P fe8846f2a6a74327bd039989735bb923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:00:48.891959 29353 raft_consensus.cc:729] T 87c881a31862415ab0fdfd9bb3c989ee P fe8846f2a6a74327bd039989735bb923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe8846f2a6a74327bd039989735bb923, State: Initialized, Role: FOLLOWER
I20260927 04:00:48.891997 29352 tablet_bootstrap.cc:654] T 87c881a31862415ab0fdfd9bb3c989ee P a584af7718a84ad2ae3b7f9d4211f873: Neither blocks nor log segments found. Creating new log.
I20260927 04:00:48.892108 29353 consensus_queue.cc:260] T 87c881a31862415ab0fdfd9bb3c989ee P fe8846f2a6a74327bd039989735bb923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9069f7895034488b852820a86b6ac57c" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 36243 } } peers { permanent_uuid: "a584af7718a84ad2ae3b7f9d4211f873" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 36119 } } peers { permanent_uuid: "fe8846f2a6a74327bd039989735bb923" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 42545 } }
I20260927 04:00:48.892326 29352 log.cc:824] T 87c881a31862415ab0fdfd9bb3c989ee P a584af7718a84ad2ae3b7f9d4211f873: Log is configured to *not* fsync() on all Append() calls
I20260927 04:00:48.892407 29319 heartbeater.cc:503] Master 127.15.185.126:38903 was elected leader, sending a full tablet report...
I20260927 04:00:48.893111 29353 ts_tablet_manager.cc:1428] T 87c881a31862415ab0fdfd9bb3c989ee P fe8846f2a6a74327bd039989735bb923: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260927 04:00:48.895458 29352 tablet_bootstrap.cc:492] T 87c881a31862415ab0fdfd9bb3c989ee P a584af7718a84ad2ae3b7f9d4211f873: No bootstrap required, opened a new log
I20260927 04:00:48.895552 29352 ts_tablet_manager.cc:1397] T 87c881a31862415ab0fdfd9bb3c989ee P a584af7718a84ad2ae3b7f9d4211f873: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260927 04:00:48.898480 29352 raft_consensus.cc:348] T 87c881a31862415ab0fdfd9bb3c989ee P a584af7718a84ad2ae3b7f9d4211f873 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9069f7895034488b852820a86b6ac57c" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 36243 } } peers { permanent_uuid: "a584af7718a84ad2ae3b7f9d4211f873" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 36119 } } peers { permanent_uuid: "fe8846f2a6a74327bd039989735bb923" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 42545 } }
I20260927 04:00:48.898658 29352 raft_consensus.cc:374] T 87c881a31862415ab0fdfd9bb3c989ee P a584af7718a84ad2ae3b7f9d4211f873 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:00:48.898702 29352 raft_consensus.cc:729] T 87c881a31862415ab0fdfd9bb3c989ee P a584af7718a84ad2ae3b7f9d4211f873 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a584af7718a84ad2ae3b7f9d4211f873, State: Initialized, Role: FOLLOWER
I20260927 04:00:48.898826 29352 consensus_queue.cc:260] T 87c881a31862415ab0fdfd9bb3c989ee P a584af7718a84ad2ae3b7f9d4211f873 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9069f7895034488b852820a86b6ac57c" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 36243 } } peers { permanent_uuid: "a584af7718a84ad2ae3b7f9d4211f873" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 36119 } } peers { permanent_uuid: "fe8846f2a6a74327bd039989735bb923" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 42545 } }
I20260927 04:00:48.899183 29352 ts_tablet_manager.cc:1428] T 87c881a31862415ab0fdfd9bb3c989ee P a584af7718a84ad2ae3b7f9d4211f873: Time spent starting tablet: real 0.004s user 0.002s sys 0.002s
I20260927 04:00:48.899329 29185 heartbeater.cc:503] Master 127.15.185.126:38903 was elected leader, sending a full tablet report...
I20260927 04:00:48.900023 29354 tablet_bootstrap.cc:492] T 87c881a31862415ab0fdfd9bb3c989ee P 9069f7895034488b852820a86b6ac57c: No bootstrap required, opened a new log
I20260927 04:00:48.900111 29354 ts_tablet_manager.cc:1397] T 87c881a31862415ab0fdfd9bb3c989ee P 9069f7895034488b852820a86b6ac57c: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.000s
I20260927 04:00:48.902647 29354 raft_consensus.cc:348] T 87c881a31862415ab0fdfd9bb3c989ee P 9069f7895034488b852820a86b6ac57c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9069f7895034488b852820a86b6ac57c" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 36243 } } peers { permanent_uuid: "a584af7718a84ad2ae3b7f9d4211f873" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 36119 } } peers { permanent_uuid: "fe8846f2a6a74327bd039989735bb923" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 42545 } }
I20260927 04:00:48.902814 29354 raft_consensus.cc:374] T 87c881a31862415ab0fdfd9bb3c989ee P 9069f7895034488b852820a86b6ac57c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:00:48.902859 29354 raft_consensus.cc:729] T 87c881a31862415ab0fdfd9bb3c989ee P 9069f7895034488b852820a86b6ac57c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9069f7895034488b852820a86b6ac57c, State: Initialized, Role: FOLLOWER
I20260927 04:00:48.902989 29354 consensus_queue.cc:260] T 87c881a31862415ab0fdfd9bb3c989ee P 9069f7895034488b852820a86b6ac57c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9069f7895034488b852820a86b6ac57c" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 36243 } } peers { permanent_uuid: "a584af7718a84ad2ae3b7f9d4211f873" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 36119 } } peers { permanent_uuid: "fe8846f2a6a74327bd039989735bb923" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 42545 } }
I20260927 04:00:48.903462 29354 ts_tablet_manager.cc:1428] T 87c881a31862415ab0fdfd9bb3c989ee P 9069f7895034488b852820a86b6ac57c: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260927 04:00:48.903615 29051 heartbeater.cc:503] Master 127.15.185.126:38903 was elected leader, sending a full tablet report...
W20260927 04:00:48.921622 29052 tablet.cc:2369] T 87c881a31862415ab0fdfd9bb3c989ee P 9069f7895034488b852820a86b6ac57c: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 04:00:48 dist-test-slave-glgz krb5kdc[27857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481648, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.126@KRBTEST.COM for HTTP/127.15.185.84@KRBTEST.COM
I20260927 04:00:49.001456 29359 raft_consensus.cc:482] T 87c881a31862415ab0fdfd9bb3c989ee P a584af7718a84ad2ae3b7f9d4211f873 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:00:49.001611 29359 raft_consensus.cc:504] T 87c881a31862415ab0fdfd9bb3c989ee P a584af7718a84ad2ae3b7f9d4211f873 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9069f7895034488b852820a86b6ac57c" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 36243 } } peers { permanent_uuid: "a584af7718a84ad2ae3b7f9d4211f873" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 36119 } } peers { permanent_uuid: "fe8846f2a6a74327bd039989735bb923" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 42545 } }
I20260927 04:00:49.002041 29359 leader_election.cc:288] T 87c881a31862415ab0fdfd9bb3c989ee P a584af7718a84ad2ae3b7f9d4211f873 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9069f7895034488b852820a86b6ac57c (127.15.185.65:36243), fe8846f2a6a74327bd039989735bb923 (127.15.185.67:42545)
I20260927 04:00:49.006462 29350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:49.002495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:48455 (local address 127.15.185.65:36243)
0927 04:00:49.002557 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 04:00:49.002559 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:00:49.002795 (+ 236us) server_negotiation.cc:407] Connection header received
0927 04:00:49.002908 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:00:49.002910 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:00:49.002920 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:00:49.002952 (+ 32us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:00:49.003227 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:49.003967 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:49.005512 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:49.006074 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:49.006122 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:49.006131 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 04:00:49.006346 (+ 215us) server_negotiation.cc:299] Negotiation successful
0927 04:00:49.006385 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 04:00:49.006714 29362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:49.002249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.65:36243 (local address 127.15.185.66:48455)
0927 04:00:49.002720 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:49.002733 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 04:00:49.002826 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:00:49.003025 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:00:49.003026 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:00:49.003027 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:00:49.003139 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:49.003142 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:49.004077 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:49.004078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:49.005420 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:49.005422 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:49.006202 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:49.006203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:49.006272 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:49.006275 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 04:00:49.006556 (+ 281us) client_negotiation.cc:240] Negotiation successful
0927 04:00:49.006604 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":122,"threads_started":1}
I20260927 04:00:49.007364 29003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87c881a31862415ab0fdfd9bb3c989ee" candidate_uuid: "a584af7718a84ad2ae3b7f9d4211f873" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9069f7895034488b852820a86b6ac57c" is_pre_election: true
I20260927 04:00:49.007557 29003 raft_consensus.cc:2454] T 87c881a31862415ab0fdfd9bb3c989ee P 9069f7895034488b852820a86b6ac57c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a584af7718a84ad2ae3b7f9d4211f873 in term 0.
I20260927 04:00:49.007612 29351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:49.002498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:38473 (local address 127.15.185.67:42545)
0927 04:00:49.002592 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0927 04:00:49.002594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:00:49.004389 (+ 1795us) server_negotiation.cc:407] Connection header received
0927 04:00:49.004487 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:00:49.004489 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:00:49.004499 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:00:49.004528 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:00:49.004710 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:49.005385 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:49.006641 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:49.007202 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:49.007278 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:49.007285 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0927 04:00:49.007504 (+ 219us) server_negotiation.cc:299] Negotiation successful
0927 04:00:49.007545 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 04:00:49.007838 29363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:49.002284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.67:42545 (local address 127.15.185.66:38473)
0927 04:00:49.004320 (+ 2036us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:49.004333 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 04:00:49.004415 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:00:49.004586 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:00:49.004587 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:00:49.004588 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:00:49.004640 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:49.004642 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:49.005482 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:49.005483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:49.006548 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:49.006550 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:49.007344 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:49.007345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:49.007418 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:49.007422 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 04:00:49.007681 (+ 259us) client_negotiation.cc:240] Negotiation successful
0927 04:00:49.007732 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2017,"thread_start_us":93,"threads_started":1}
I20260927 04:00:49.008121 29074 leader_election.cc:302] T 87c881a31862415ab0fdfd9bb3c989ee P a584af7718a84ad2ae3b7f9d4211f873 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9069f7895034488b852820a86b6ac57c, a584af7718a84ad2ae3b7f9d4211f873; no voters:
I20260927 04:00:49.008260 29359 raft_consensus.cc:2781] T 87c881a31862415ab0fdfd9bb3c989ee P a584af7718a84ad2ae3b7f9d4211f873 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 04:00:49.008213 29273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87c881a31862415ab0fdfd9bb3c989ee" candidate_uuid: "a584af7718a84ad2ae3b7f9d4211f873" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe8846f2a6a74327bd039989735bb923" is_pre_election: true
I20260927 04:00:49.008302 29359 raft_consensus.cc:482] T 87c881a31862415ab0fdfd9bb3c989ee P a584af7718a84ad2ae3b7f9d4211f873 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 04:00:49.008325 29359 raft_consensus.cc:3037] T 87c881a31862415ab0fdfd9bb3c989ee P a584af7718a84ad2ae3b7f9d4211f873 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:00:49.008365 29273 raft_consensus.cc:2454] T 87c881a31862415ab0fdfd9bb3c989ee P fe8846f2a6a74327bd039989735bb923 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a584af7718a84ad2ae3b7f9d4211f873 in term 0.
I20260927 04:00:49.009268 29359 raft_consensus.cc:504] T 87c881a31862415ab0fdfd9bb3c989ee P a584af7718a84ad2ae3b7f9d4211f873 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9069f7895034488b852820a86b6ac57c" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 36243 } } peers { permanent_uuid: "a584af7718a84ad2ae3b7f9d4211f873" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 36119 } } peers { permanent_uuid: "fe8846f2a6a74327bd039989735bb923" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 42545 } }
I20260927 04:00:49.009406 29359 leader_election.cc:288] T 87c881a31862415ab0fdfd9bb3c989ee P a584af7718a84ad2ae3b7f9d4211f873 [CANDIDATE]: Term 1 election: Requested vote from peers 9069f7895034488b852820a86b6ac57c (127.15.185.65:36243), fe8846f2a6a74327bd039989735bb923 (127.15.185.67:42545)
I20260927 04:00:49.009578 29273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87c881a31862415ab0fdfd9bb3c989ee" candidate_uuid: "a584af7718a84ad2ae3b7f9d4211f873" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe8846f2a6a74327bd039989735bb923"
I20260927 04:00:49.009652 29273 raft_consensus.cc:3037] T 87c881a31862415ab0fdfd9bb3c989ee P fe8846f2a6a74327bd039989735bb923 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:00:49.010550 29003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87c881a31862415ab0fdfd9bb3c989ee" candidate_uuid: "a584af7718a84ad2ae3b7f9d4211f873" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9069f7895034488b852820a86b6ac57c"
I20260927 04:00:49.010635 29003 raft_consensus.cc:3037] T 87c881a31862415ab0fdfd9bb3c989ee P 9069f7895034488b852820a86b6ac57c [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:00:49.011590 29273 raft_consensus.cc:2454] T 87c881a31862415ab0fdfd9bb3c989ee P fe8846f2a6a74327bd039989735bb923 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a584af7718a84ad2ae3b7f9d4211f873 in term 1.
I20260927 04:00:49.011957 29075 leader_election.cc:302] T 87c881a31862415ab0fdfd9bb3c989ee P a584af7718a84ad2ae3b7f9d4211f873 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a584af7718a84ad2ae3b7f9d4211f873, fe8846f2a6a74327bd039989735bb923; no voters:
I20260927 04:00:49.012068 29359 raft_consensus.cc:2781] T 87c881a31862415ab0fdfd9bb3c989ee P a584af7718a84ad2ae3b7f9d4211f873 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:00:49.012274 29359 raft_consensus.cc:686] T 87c881a31862415ab0fdfd9bb3c989ee P a584af7718a84ad2ae3b7f9d4211f873 [term 1 LEADER]: Becoming Leader. State: Replica: a584af7718a84ad2ae3b7f9d4211f873, State: Running, Role: LEADER
I20260927 04:00:49.012375 29359 consensus_queue.cc:237] T 87c881a31862415ab0fdfd9bb3c989ee P a584af7718a84ad2ae3b7f9d4211f873 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9069f7895034488b852820a86b6ac57c" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 36243 } } peers { permanent_uuid: "a584af7718a84ad2ae3b7f9d4211f873" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 36119 } } peers { permanent_uuid: "fe8846f2a6a74327bd039989735bb923" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 42545 } }
I20260927 04:00:49.012725 29003 raft_consensus.cc:2454] T 87c881a31862415ab0fdfd9bb3c989ee P 9069f7895034488b852820a86b6ac57c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a584af7718a84ad2ae3b7f9d4211f873 in term 1.
I20260927 04:00:49.013389 28846 catalog_manager.cc:5495] T 87c881a31862415ab0fdfd9bb3c989ee P a584af7718a84ad2ae3b7f9d4211f873 reported cstate change: term changed from 0 to 1, leader changed from <none> to a584af7718a84ad2ae3b7f9d4211f873 (127.15.185.66). New cstate: current_term: 1 leader_uuid: "a584af7718a84ad2ae3b7f9d4211f873" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9069f7895034488b852820a86b6ac57c" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 36243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a584af7718a84ad2ae3b7f9d4211f873" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 36119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe8846f2a6a74327bd039989735bb923" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 42545 } health_report { overall_health: UNKNOWN } } }
I20260927 04:00:49.024170 29349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:49.020683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52670 (local address 127.15.185.66:36119)
0927 04:00:49.020727 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 04:00:49.020729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:00:49.020738 (+ 9us) server_negotiation.cc:407] Connection header received
0927 04:00:49.020777 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:00:49.020778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:00:49.020811 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:00:49.020886 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:00:49.021083 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:49.021874 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:49.023009 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:49.023352 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:49.023412 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:49.023606 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:00:49.023741 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:00:49.023826 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 04:00:49.024064 (+ 238us) server_negotiation.cc:299] Negotiation successful
0927 04:00:49.024085 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 04:00:49.029201 29273 raft_consensus.cc:1261] T 87c881a31862415ab0fdfd9bb3c989ee P fe8846f2a6a74327bd039989735bb923 [term 1 FOLLOWER]: Refusing update from remote peer a584af7718a84ad2ae3b7f9d4211f873: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:00:49.029229 29003 raft_consensus.cc:1261] T 87c881a31862415ab0fdfd9bb3c989ee P 9069f7895034488b852820a86b6ac57c [term 1 FOLLOWER]: Refusing update from remote peer a584af7718a84ad2ae3b7f9d4211f873: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:00:49.029579 29359 consensus_queue.cc:1035] T 87c881a31862415ab0fdfd9bb3c989ee P a584af7718a84ad2ae3b7f9d4211f873 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069f7895034488b852820a86b6ac57c" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 36243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:00:49.029733 29364 consensus_queue.cc:1035] T 87c881a31862415ab0fdfd9bb3c989ee P a584af7718a84ad2ae3b7f9d4211f873 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe8846f2a6a74327bd039989735bb923" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 42545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:00:49.034116 29367 mvcc.cc:205] Tried to move back new op lower bound from 7333812834420371456 to 7333812834355134464. Current Snapshot: MvccSnapshot[applied={T|T < 7333812834420371456}]
I20260927 04:00:49.035430 29369 mvcc.cc:205] Tried to move back new op lower bound from 7333812834420371456 to 7333812834355134464. Current Snapshot: MvccSnapshot[applied={T|T < 7333812834420371456}]
I20260927 04:00:49.039170 28846 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52640:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:00:49.039270 28846 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52640:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:00:49.040022 28846 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aff4fb0709a14bfbbc2e9060c6d43b72: Sending DeleteTablet for 3 replicas of tablet 87c881a31862415ab0fdfd9bb3c989ee
I20260927 04:00:49.040414 29119 tablet_service.cc:1514] Processing DeleteTablet for tablet 87c881a31862415ab0fdfd9bb3c989ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:00:49 UTC) from {username='kudu', principal='kudu/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:52654
I20260927 04:00:49.040723 29253 tablet_service.cc:1514] Processing DeleteTablet for tablet 87c881a31862415ab0fdfd9bb3c989ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:00:49 UTC) from {username='kudu', principal='kudu/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:51560
I20260927 04:00:49.040735 28985 tablet_service.cc:1514] Processing DeleteTablet for tablet 87c881a31862415ab0fdfd9bb3c989ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:00:49 UTC) from {username='kudu', principal='kudu/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:37032
I20260927 04:00:49.041654 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 28891
I20260927 04:00:49.047222 29378 tablet_replica.cc:330] T 87c881a31862415ab0fdfd9bb3c989ee P fe8846f2a6a74327bd039989735bb923: stopping tablet replica
I20260927 04:00:49.047379 29377 tablet_replica.cc:330] T 87c881a31862415ab0fdfd9bb3c989ee P a584af7718a84ad2ae3b7f9d4211f873: stopping tablet replica
I20260927 04:00:49.047513 29377 raft_consensus.cc:2229] T 87c881a31862415ab0fdfd9bb3c989ee P a584af7718a84ad2ae3b7f9d4211f873 [term 1 LEADER]: Raft consensus shutting down.
I20260927 04:00:49.047637 29378 raft_consensus.cc:2229] T 87c881a31862415ab0fdfd9bb3c989ee P fe8846f2a6a74327bd039989735bb923 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:00:49.047825 29377 raft_consensus.cc:2258] T 87c881a31862415ab0fdfd9bb3c989ee P a584af7718a84ad2ae3b7f9d4211f873 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:00:49.047977 29378 raft_consensus.cc:2258] T 87c881a31862415ab0fdfd9bb3c989ee P fe8846f2a6a74327bd039989735bb923 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:00:49.048694 29378 ts_tablet_manager.cc:1905] T 87c881a31862415ab0fdfd9bb3c989ee P fe8846f2a6a74327bd039989735bb923: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:00:49.049118 29377 ts_tablet_manager.cc:1905] T 87c881a31862415ab0fdfd9bb3c989ee P a584af7718a84ad2ae3b7f9d4211f873: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:00:49.054093 29378 ts_tablet_manager.cc:1918] T 87c881a31862415ab0fdfd9bb3c989ee P fe8846f2a6a74327bd039989735bb923: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:00:49.056319 29377 ts_tablet_manager.cc:1918] T 87c881a31862415ab0fdfd9bb3c989ee P a584af7718a84ad2ae3b7f9d4211f873: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:00:49.057222 29378 log.cc:1196] T 87c881a31862415ab0fdfd9bb3c989ee P fe8846f2a6a74327bd039989735bb923: Deleting WAL directory at /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-2/wal/wals/87c881a31862415ab0fdfd9bb3c989ee
I20260927 04:00:49.057652 29378 ts_tablet_manager.cc:1939] T 87c881a31862415ab0fdfd9bb3c989ee P fe8846f2a6a74327bd039989735bb923: Deleting consensus metadata
I20260927 04:00:49.059973 29377 log.cc:1196] T 87c881a31862415ab0fdfd9bb3c989ee P a584af7718a84ad2ae3b7f9d4211f873: Deleting WAL directory at /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481498018407-16101-0/minicluster-data/ts-1/wal/wals/87c881a31862415ab0fdfd9bb3c989ee
I20260927 04:00:49.062541 28840 catalog_manager.cc:4841] TS fe8846f2a6a74327bd039989735bb923 (127.15.185.67:42545): tablet 87c881a31862415ab0fdfd9bb3c989ee (table test-table [id=43535011f96f44c3bd97bce2af6ccd2d]) successfully deleted
W20260927 04:00:49.062963 28839 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.185.65:36243 (error 108)
I20260927 04:00:49.063946 29348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:49.063783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.65:36243 (local address 127.0.0.1:37046)
0927 04:00:49.063850 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:49.063907 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.65:36243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 04:00:49.063951 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 29053
W20260927 04:00:49.065918 28839 catalog_manager.cc:4584] TS 9069f7895034488b852820a86b6ac57c (127.15.185.65:36243): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 87c881a31862415ab0fdfd9bb3c989ee: Network error: Client connection negotiation failed: client connection to 127.15.185.65:36243: connect: Connection refused (error 111)
W20260927 04:00:49.071022 29348 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:00:49.070820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.66:36119 (local address 127.0.0.1:52682)
0927 04:00:49.070887 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:49.070901 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 04:00:49.070954 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.66:36119: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 04:00:49.071350 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 29188
I20260927 04:00:49.079653 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 28820
I20260927 04:00:49.089131 16101 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 04:00:49.433593 16101 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 04:00:49.433718 16101 mini_ranger.cc:66] Stopping Ranger...
I20260927 04:00:49.778292 16101 mini_ranger.cc:68] Stopped Ranger
2026-09-27 04:00:49.779 UTC [27887] LOG: received smart shutdown request
2026-09-27 04:00:49.784 UTC [27887] LOG: background worker "logical replication launcher" (PID 27896) exited with exit code 1
2026-09-27 04:00:49.784 UTC [27891] LOG: shutting down
2026-09-27 04:00:49.968 UTC [27887] LOG: database system is shut down
2026-09-27T04:00:49Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48051 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:00:50 dist-test-slave-glgz krb5kdc[29390](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:00:50 dist-test-slave-glgz krb5kdc[29390](info): set up 2 sockets
Sep 27 04:00:50 dist-test-slave-glgz krb5kdc[29390](info): commencing operation
krb5kdc: starting...
W20260927 04:00:51.115393 16101 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Sep 27 04:00:51 dist-test-slave-glgz krb5kdc[29390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:00:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:00:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.185.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab.
Entry for principal kudu/127.15.185.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab.
I20260927 04:00:51.234799 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:45857
--webserver_interface=127.15.185.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:44547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.185.126:45857,127.15.185.125:35339,127.15.185.124:43821
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:00:51.315117 29403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:00:51.315398 29403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:00:51.315438 29403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:00:51.316666 29403 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:00:51.316715 29403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:00:51.316728 29403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:00:51.316741 29403 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:00:51.316753 29403 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:00:51.318233 29403 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:44547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.185.126:45857,127.15.185.125:35339,127.15.185.124:43821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:45857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:00:51.318441 29403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:00:51.318619 29403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:00:51.323421 29411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:00:51.323423 29408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:00:51.323551 29409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:00:51.323863 29403 server_base.cc:1034] running on GCE node
I20260927 04:00:51.324117 29403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:00:51.324422 29403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:00:51.325572 29403 hybrid_clock.cc:648] HybridClock initialized: now 1790481651325558 us; error 28 us; skew 500 ppm
Sep 27 04:00:51 dist-test-slave-glgz krb5kdc[29390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481651, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:00:51.332741 29403 init.cc:377] Logged in from keytab as kudu/127.15.185.126@KRBTEST.COM (short username kudu)
I20260927 04:00:51.333228 29403 webserver.cc:460] Webserver started at http://127.15.185.126:33235/ using document root <none> and password file <none>
I20260927 04:00:51.333482 29403 fs_manager.cc:359] Metadata directory not provided
I20260927 04:00:51.333547 29403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:00:51.333711 29403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:00:51.334818 29403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/minicluster-data/master-0/data/instance:
uuid: "f6cb4bf4a3554a9ab3a78dfe25853f0d"
format_stamp: "Formatted at 2026-09-27 04:00:51 on dist-test-slave-glgz"
server_key: "dc05dd1d618ec61391d0a788a19e1ac1"
server_key_iv: "beef564bd679881dc1803a6a7f5d69a1"
server_key_version: "encryptionkey@0"
I20260927 04:00:51.335212 29403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/minicluster-data/master-0/wal/instance:
uuid: "f6cb4bf4a3554a9ab3a78dfe25853f0d"
format_stamp: "Formatted at 2026-09-27 04:00:51 on dist-test-slave-glgz"
server_key: "dc05dd1d618ec61391d0a788a19e1ac1"
server_key_iv: "beef564bd679881dc1803a6a7f5d69a1"
server_key_version: "encryptionkey@0"
I20260927 04:00:51.336577 29403 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:00:51.337378 29418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:00:51.337596 29403 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:00:51.337750 29403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/minicluster-data/master-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/minicluster-data/master-0/wal
uuid: "f6cb4bf4a3554a9ab3a78dfe25853f0d"
format_stamp: "Formatted at 2026-09-27 04:00:51 on dist-test-slave-glgz"
server_key: "dc05dd1d618ec61391d0a788a19e1ac1"
server_key_iv: "beef564bd679881dc1803a6a7f5d69a1"
server_key_version: "encryptionkey@0"
I20260927 04:00:51.337832 29403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:00:51.352061 29403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:00:51.353981 29403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:00:51.354146 29403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:00:51.357995 29403 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.126:45857
I20260927 04:00:51.358049 29470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.126:45857 every 8 connection(s)
I20260927 04:00:51.358408 29403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
I20260927 04:00:51.359292 29471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:00:51.359727 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 29403
I20260927 04:00:51.359853 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/minicluster-data/master-0/wal/instance
I20260927 04:00:51.360056 16101 external_mini_cluster.cc:1428] Setting key f62ff7374ba4ec39bbfa8da28bb430eb
I20260927 04:00:51.360975 29471 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45857 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 04:00:51 dist-test-slave-glgz krb5kdc[29390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481651, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.126@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.185.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.125@KRBTEST.COM" created.
I20260927 04:00:51.370939 29474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:51.361800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45857 (local address 127.0.0.1:42782)
0927 04:00:51.361970 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:51.362014 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 04:00:51.362508 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:00:51.362796 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:00:51.362799 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:00:51.363131 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:00:51.363282 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:51.363287 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:51.364353 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:51.364354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:51.365035 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:51.365037 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:51.365540 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:51.365541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:51.365604 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:51.366181 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:00:51.366189 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:00:51.367567 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:00:51.369328 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:00:51.369333 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:00:51.369335 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:00:51.369523 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:00:51.369781 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:00:51.369782 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:00:51.369783 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:00:51.369918 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:00:51.370338 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:00:51.370344 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:00:51.370404 (+ 60us) client_negotiation.cc:769] Sending connection context
0927 04:00:51.370482 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 04:00:51.370608 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":97,"threads_started":1}
I20260927 04:00:51.370945 29475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:51.362020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42782 (local address 127.15.185.126:45857)
0927 04:00:51.362156 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 04:00:51.362160 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:00:51.362188 (+ 28us) server_negotiation.cc:407] Connection header received
0927 04:00:51.362635 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:00:51.362637 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:00:51.362653 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:00:51.362790 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:00:51.363390 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:51.364234 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:51.365132 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:51.365449 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:51.365570 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:51.367660 (+ 2090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:00:51.367666 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:00:51.367667 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:00:51.367677 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:00:51.369195 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:51.369611 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:51.369614 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:51.369614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:51.369693 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:51.369997 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:51.369998 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:51.369998 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:51.370260 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:00:51.370352 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 04:00:51.370538 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 04:00:51.370702 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":109,"thread_start_us":72,"threads_started":1}
I20260927 04:00:51.371781 29471 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.185.125" port: 35339 } has no permanent_uuid. Determining permanent_uuid...
I20260927 04:00:51.372200 29474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:51.372040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.125:35339 (local address 127.0.0.1:45430)
0927 04:00:51.372131 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:51.372179 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.125:35339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260927 04:00:51.372380 29419 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.185.125:35339: connect: Connection refused (error 111)
I20260927 04:00:51.372786 29474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:51.372672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.125:35339 (local address 127.0.0.1:45436)
0927 04:00:51.372726 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:51.372756 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.125:35339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 04:00:51.372913 29471 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.185.125:35339: Network error: Client connection negotiation failed: client connection to 127.15.185.125:35339: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.125.keytab.
Entry for principal kudu/127.15.185.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.125.keytab.
I20260927 04:00:51.386560 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.125:35339
--webserver_interface=127.15.185.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.125.keytab
--principal=kudu/127.15.185.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:44547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.185.126:45857,127.15.185.125:35339,127.15.185.124:43821
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 04:00:51.422158 29471 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.185.125" port: 35339 } attempt: 1
I20260927 04:00:51.422717 29474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:51.422547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.125:35339 (local address 127.0.0.1:45444)
0927 04:00:51.422619 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:51.422675 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.125:35339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 04:00:51.423173 29474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:51.423064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.125:35339 (local address 127.0.0.1:45448)
0927 04:00:51.423110 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:51.423143 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.125:35339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 04:00:51.423272 29471 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.185.125:35339: Network error: Client connection negotiation failed: client connection to 127.15.185.125:35339: connect: Connection refused (error 111)
W20260927 04:00:51.472695 29478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:00:51.472896 29478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:00:51.472918 29478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:00:51.473996 29478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:00:51.474032 29478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:00:51.474046 29478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:00:51.474059 29478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:00:51.474071 29478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:00:51.475526 29478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:44547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.15.185.126:45857,127.15.185.125:35339,127.15.185.124:43821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.125.keytab
--principal=kudu/127.15.185.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.125:35339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.15.185.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:00:51.475725 29478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:00:51.475901 29478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:00:51.480814 29483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:00:51.480808 29484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:00:51.480808 29486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:00:51.480825 29478 server_base.cc:1034] running on GCE node
I20260927 04:00:51.481348 29478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:00:51.481662 29478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:00:51.482842 29478 hybrid_clock.cc:648] HybridClock initialized: now 1790481651482810 us; error 50 us; skew 500 ppm
Sep 27 04:00:51 dist-test-slave-glgz krb5kdc[29390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481651, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:00:51.485507 29478 init.cc:377] Logged in from keytab as kudu/127.15.185.125@KRBTEST.COM (short username kudu)
I20260927 04:00:51.485898 29478 webserver.cc:460] Webserver started at http://127.15.185.125:41419/ using document root <none> and password file <none>
I20260927 04:00:51.486090 29478 fs_manager.cc:359] Metadata directory not provided
I20260927 04:00:51.486130 29478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:00:51.486218 29478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:00:51.487319 29478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/minicluster-data/master-1/data/instance:
uuid: "98fa76cd38b24a048dd743e2b43cef62"
format_stamp: "Formatted at 2026-09-27 04:00:51 on dist-test-slave-glgz"
server_key: "2fb55a94f2371b7c6e35637caa531340"
server_key_iv: "a954f8ae9cbbb37939160cd912472dc8"
server_key_version: "encryptionkey@0"
I20260927 04:00:51.487623 29478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/minicluster-data/master-1/wal/instance:
uuid: "98fa76cd38b24a048dd743e2b43cef62"
format_stamp: "Formatted at 2026-09-27 04:00:51 on dist-test-slave-glgz"
server_key: "2fb55a94f2371b7c6e35637caa531340"
server_key_iv: "a954f8ae9cbbb37939160cd912472dc8"
server_key_version: "encryptionkey@0"
I20260927 04:00:51.488940 29478 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:00:51.489610 29493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:00:51.489830 29478 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:00:51.489925 29478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/minicluster-data/master-1/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/minicluster-data/master-1/wal
uuid: "98fa76cd38b24a048dd743e2b43cef62"
format_stamp: "Formatted at 2026-09-27 04:00:51 on dist-test-slave-glgz"
server_key: "2fb55a94f2371b7c6e35637caa531340"
server_key_iv: "a954f8ae9cbbb37939160cd912472dc8"
server_key_version: "encryptionkey@0"
I20260927 04:00:51.489982 29478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:00:51.491401 29471 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.185.125" port: 35339 } attempt: 2
I20260927 04:00:51.491775 29474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:51.491649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.125:35339 (local address 127.0.0.1:45452)
0927 04:00:51.491705 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:51.491738 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.125:35339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 04:00:51.492144 29474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:51.492050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.125:35339 (local address 127.0.0.1:45454)
0927 04:00:51.492097 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:51.492127 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.125:35339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 04:00:51.492246 29471 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.185.125:35339: Network error: Client connection negotiation failed: client connection to 127.15.185.125:35339: connect: Connection refused (error 111)
I20260927 04:00:51.499382 29478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:00:51.501287 29478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:00:51.501403 29478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:00:51.505354 29478 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.125:35339
I20260927 04:00:51.505419 29545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.125:35339 every 8 connection(s)
I20260927 04:00:51.505757 29478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/minicluster-data/master-1/data/info.pb
I20260927 04:00:51.506634 29546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:00:51.508286 29546 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45857 } has no permanent_uuid. Determining permanent_uuid...
I20260927 04:00:51.511502 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 29478
I20260927 04:00:51.511596 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/minicluster-data/master-1/wal/instance
I20260927 04:00:51.511868 16101 external_mini_cluster.cc:1428] Setting key 059f70bed81d3156441f49568079396a
Sep 27 04:00:51 dist-test-slave-glgz krb5kdc[29390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481651, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.125@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 04:00:51.518293 29475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:51.509120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42784 (local address 127.15.185.126:45857)
0927 04:00:51.509198 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0927 04:00:51.509200 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:00:51.509443 (+ 243us) server_negotiation.cc:407] Connection header received
0927 04:00:51.509990 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:00:51.509991 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:00:51.510018 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:00:51.510071 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:00:51.510739 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:51.511283 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:51.512432 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:51.512636 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:51.512723 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:51.515288 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:00:51.515290 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:00:51.515290 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:00:51.515299 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:00:51.516862 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:51.517300 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:51.517301 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:51.517302 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:51.517363 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:51.517616 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:51.517617 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:51.517618 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:51.517743 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:00:51.517817 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 04:00:51.518096 (+ 279us) server_negotiation.cc:299] Negotiation successful
0927 04:00:51.518189 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 04:00:51.518492 29548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:51.509103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45857 (local address 127.0.0.1:42784)
0927 04:00:51.509314 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:51.509356 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 04:00:51.509869 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:00:51.510152 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:00:51.510155 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:00:51.510434 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:00:51.510636 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:51.510641 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:51.511390 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:51.511392 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:51.512343 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:51.512346 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:51.512740 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:51.512741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:51.512906 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:51.513483 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:00:51.513494 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:00:51.515176 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:00:51.516963 (+ 1787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:00:51.516971 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:00:51.516974 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:00:51.517210 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:00:51.517430 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:00:51.517431 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:00:51.517431 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:00:51.517548 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:00:51.517808 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:00:51.517815 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:00:51.517965 (+ 150us) client_negotiation.cc:769] Sending connection context
0927 04:00:51.518074 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 04:00:51.518239 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":93,"threads_started":1}
I20260927 04:00:51.519078 29546 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.185.125" port: 35339 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.15.185.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.124@KRBTEST.COM" created.
Sep 27 04:00:51 dist-test-slave-glgz krb5kdc[29390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481651, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.125@KRBTEST.COM for kudu/127.15.185.125@KRBTEST.COM
I20260927 04:00:51.527001 29548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:51.519382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.125:35339 (local address 127.0.0.1:45456)
0927 04:00:51.519436 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:51.519447 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 04:00:51.519488 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:00:51.519800 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:00:51.519800 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:00:51.519946 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:00:51.519992 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:51.519993 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:51.520931 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:51.520932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:51.521486 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:51.521488 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:51.522009 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:51.522010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:51.522076 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:51.522560 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:00:51.522565 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:00:51.523830 (+ 1265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:00:51.525929 (+ 2099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:00:51.525930 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:00:51.525931 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:00:51.526092 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:00:51.526311 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:00:51.526312 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:00:51.526313 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:00:51.526369 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:00:51.526651 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:00:51.526652 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:00:51.526690 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 04:00:51.526756 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 04:00:51.526859 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 04:00:51.527061 29550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:51.519458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45456 (local address 127.15.185.125:35339)
0927 04:00:51.519657 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 04:00:51.519660 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:00:51.519677 (+ 17us) server_negotiation.cc:407] Connection header received
0927 04:00:51.519693 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:00:51.519694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:00:51.519712 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:00:51.519799 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:00:51.520061 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:51.520844 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:51.521567 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:51.521923 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:51.522002 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:51.523918 (+ 1916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:00:51.523926 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:00:51.523927 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:00:51.523933 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:00:51.525831 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:51.526178 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:51.526182 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:51.526183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:51.526248 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:51.526433 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:51.526435 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:51.526435 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:51.526582 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:00:51.526673 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 04:00:51.526783 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 04:00:51.526915 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":165,"thread_start_us":130,"threads_started":1}
I20260927 04:00:51.527706 29546 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.185.124" port: 43821 } has no permanent_uuid. Determining permanent_uuid...
I20260927 04:00:51.528160 29548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:51.528021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.124:43821 (local address 127.0.0.1:34330)
0927 04:00:51.528073 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:51.528133 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.124:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260927 04:00:51.528311 29494 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.185.124:43821: connect: Connection refused (error 111)
I20260927 04:00:51.528764 29548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:51.528657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.124:43821 (local address 127.0.0.1:34338)
0927 04:00:51.528709 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:51.528740 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.124:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 04:00:51.528913 29546 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.185.124:43821: Network error: Client connection negotiation failed: client connection to 127.15.185.124:43821: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.124.keytab.
Entry for principal kudu/127.15.185.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.124.keytab.
I20260927 04:00:51.538446 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.124:43821
--webserver_interface=127.15.185.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.124.keytab
--principal=kudu/127.15.185.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:44547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.185.126:45857,127.15.185.125:35339,127.15.185.124:43821
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 04:00:51.578151 29546 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.185.124" port: 43821 } attempt: 1
I20260927 04:00:51.578689 29548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:51.578547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.124:43821 (local address 127.0.0.1:34340)
0927 04:00:51.578605 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:51.578654 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.124:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 04:00:51.579110 29548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:51.578964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.124:43821 (local address 127.0.0.1:34346)
0927 04:00:51.579014 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:51.579080 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.124:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 04:00:51.579229 29546 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.185.124:43821: Network error: Client connection negotiation failed: client connection to 127.15.185.124:43821: connect: Connection refused (error 111)
I20260927 04:00:51.583395 29471 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.185.125" port: 35339 } attempt: 3
Sep 27 04:00:51 dist-test-slave-glgz krb5kdc[29390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481651, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.126@KRBTEST.COM for kudu/127.15.185.125@KRBTEST.COM
I20260927 04:00:51.590291 29474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:51.583723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.125:35339 (local address 127.0.0.1:45466)
0927 04:00:51.583782 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:51.583794 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 04:00:51.583849 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:00:51.584002 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:00:51.584003 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:00:51.584175 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:00:51.584242 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:51.584244 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:51.584928 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:51.584929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:51.585430 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:51.585431 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:51.585837 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:51.585839 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:51.585919 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:51.586412 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:00:51.586417 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:00:51.587650 (+ 1233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:00:51.589253 (+ 1603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:00:51.589254 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:00:51.589254 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:00:51.589427 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:00:51.589641 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:00:51.589642 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:00:51.589643 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:00:51.589712 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:00:51.589952 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:00:51.589953 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:00:51.590003 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 04:00:51.590084 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 04:00:51.590188 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 04:00:51.590289 29550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:51.583781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45466 (local address 127.15.185.125:35339)
0927 04:00:51.583829 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 04:00:51.583831 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:00:51.583844 (+ 13us) server_negotiation.cc:407] Connection header received
0927 04:00:51.583918 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:00:51.583919 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:00:51.583931 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:00:51.583996 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:00:51.584335 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:51.584848 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:51.585524 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:51.585750 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:51.585828 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:51.587746 (+ 1918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:00:51.587748 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:00:51.587748 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:00:51.587757 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:00:51.589144 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:51.589519 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:51.589520 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:51.589521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:51.589578 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:51.589773 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:51.589775 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:51.589775 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:51.589888 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:00:51.589961 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 04:00:51.590111 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 04:00:51.590193 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 04:00:51.590672 29471 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.185.124" port: 43821 } has no permanent_uuid. Determining permanent_uuid...
I20260927 04:00:51.591015 29474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:51.590917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.124:43821 (local address 127.0.0.1:34356)
0927 04:00:51.590963 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:51.590991 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.124:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 04:00:51.591379 29474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:51.591300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.124:43821 (local address 127.0.0.1:34368)
0927 04:00:51.591341 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:51.591365 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.124:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 04:00:51.591470 29471 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.185.124:43821: Network error: Client connection negotiation failed: client connection to 127.15.185.124:43821: connect: Connection refused (error 111)
W20260927 04:00:51.621965 29553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:00:51.622184 29553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:00:51.622208 29553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 04:00:51.622598 29471 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.185.124" port: 43821 } attempt: 1
I20260927 04:00:51.623118 29474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:51.622948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.124:43821 (local address 127.0.0.1:34384)
0927 04:00:51.623009 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:51.623080 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.124:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 04:00:51.623533 29474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:51.623404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.124:43821 (local address 127.0.0.1:34392)
0927 04:00:51.623473 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:51.623507 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.124:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 04:00:51.623607 29553 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:00:51.623677 29553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:00:51.623718 29553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:00:51.623749 29553 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:00:51.623775 29553 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 04:00:51.623806 29471 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.185.124:43821: Network error: Client connection negotiation failed: client connection to 127.15.185.124:43821: connect: Connection refused (error 111)
I20260927 04:00:51.625314 29553 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:44547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.15.185.126:45857,127.15.185.125:35339,127.15.185.124:43821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.124.keytab
--principal=kudu/127.15.185.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.124:43821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.15.185.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:00:51.625569 29553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:00:51.625810 29553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:00:51.630508 29561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:00:51.630535 29558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:00:51.630586 29559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:00:51.630748 29553 server_base.cc:1034] running on GCE node
I20260927 04:00:51.630973 29553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:00:51.631276 29553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:00:51.632436 29553 hybrid_clock.cc:648] HybridClock initialized: now 1790481651632425 us; error 28 us; skew 500 ppm
Sep 27 04:00:51 dist-test-slave-glgz krb5kdc[29390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481651, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:00:51.635135 29553 init.cc:377] Logged in from keytab as kudu/127.15.185.124@KRBTEST.COM (short username kudu)
I20260927 04:00:51.635612 29553 webserver.cc:460] Webserver started at http://127.15.185.124:39147/ using document root <none> and password file <none>
I20260927 04:00:51.635838 29553 fs_manager.cc:359] Metadata directory not provided
I20260927 04:00:51.635902 29553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:00:51.636050 29553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:00:51.637144 29553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/minicluster-data/master-2/data/instance:
uuid: "fe29f070521340928ed3b785ca16c139"
format_stamp: "Formatted at 2026-09-27 04:00:51 on dist-test-slave-glgz"
server_key: "ba7289d61444798185f387a5ccb10339"
server_key_iv: "1a72a31d1b1c8c7f1751a1230a28ffd6"
server_key_version: "encryptionkey@0"
I20260927 04:00:51.637496 29553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/minicluster-data/master-2/wal/instance:
uuid: "fe29f070521340928ed3b785ca16c139"
format_stamp: "Formatted at 2026-09-27 04:00:51 on dist-test-slave-glgz"
server_key: "ba7289d61444798185f387a5ccb10339"
server_key_iv: "1a72a31d1b1c8c7f1751a1230a28ffd6"
server_key_version: "encryptionkey@0"
I20260927 04:00:51.638823 29553 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 04:00:51.639606 29568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:00:51.639781 29553 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 04:00:51.639878 29553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/minicluster-data/master-2/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/minicluster-data/master-2/wal
uuid: "fe29f070521340928ed3b785ca16c139"
format_stamp: "Formatted at 2026-09-27 04:00:51 on dist-test-slave-glgz"
server_key: "ba7289d61444798185f387a5ccb10339"
server_key_iv: "1a72a31d1b1c8c7f1751a1230a28ffd6"
server_key_version: "encryptionkey@0"
I20260927 04:00:51.639958 29553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:00:51.647406 29553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:00:51.647372 29546 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.185.124" port: 43821 } attempt: 2
I20260927 04:00:51.647814 29548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:51.647698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.124:43821 (local address 127.0.0.1:34404)
0927 04:00:51.647747 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:51.647781 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.124:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260927 04:00:51.648137 29548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:51.648076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.124:43821 (local address 127.0.0.1:34410)
0927 04:00:51.648100 (+ 24us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:51.648122 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.124:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260927 04:00:51.648255 29546 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.185.124:43821: Network error: Client connection negotiation failed: client connection to 127.15.185.124:43821: connect: Connection refused (error 111)
I20260927 04:00:51.649866 29553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:00:51.649978 29553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:00:51.653775 29553 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.124:43821
I20260927 04:00:51.653839 29620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.124:43821 every 8 connection(s)
I20260927 04:00:51.654202 29553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/minicluster-data/master-2/data/info.pb
I20260927 04:00:51.655141 29621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:00:51.656683 29621 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45857 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 04:00:51 dist-test-slave-glgz krb5kdc[29390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481651, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.124@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 04:00:51.663594 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 29553
I20260927 04:00:51.663725 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481498018407-16101-0/minicluster-data/master-2/wal/instance
I20260927 04:00:51.663928 16101 external_mini_cluster.cc:1428] Setting key 9058a3fc3e6e53abafd9ad8fe69b2913
I20260927 04:00:51.665699 29475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:51.657372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42798 (local address 127.15.185.126:45857)
0927 04:00:51.657447 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0927 04:00:51.657450 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:00:51.657633 (+ 183us) server_negotiation.cc:407] Connection header received
0927 04:00:51.658168 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:00:51.658169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:00:51.658182 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:00:51.658235 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:00:51.658869 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:51.659430 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:51.660431 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:51.660666 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:51.660751 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:51.662810 (+ 2059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:00:51.662812 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:00:51.662812 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:00:51.662821 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:00:51.664328 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:51.664751 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:51.664752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:51.664753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:51.664820 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:51.665062 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:51.665063 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:51.665063 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:51.665187 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:00:51.665256 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 04:00:51.665523 (+ 267us) server_negotiation.cc:299] Negotiation successful
0927 04:00:51.665604 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 04:00:51.665904 29623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:51.657337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45857 (local address 127.0.0.1:42798)
0927 04:00:51.657516 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:51.657553 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 04:00:51.658031 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:00:51.658307 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:00:51.658311 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:00:51.658555 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:00:51.658773 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:51.658778 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:51.659534 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:51.659539 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:51.660326 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:51.660328 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:51.660759 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:51.660760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:51.660891 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:51.661407 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:00:51.661416 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:00:51.662715 (+ 1299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:00:51.664425 (+ 1710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:00:51.664430 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:00:51.664433 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:00:51.664664 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:00:51.664883 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:00:51.664884 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:00:51.664884 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:00:51.664995 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:00:51.665258 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:00:51.665265 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:00:51.665399 (+ 134us) client_negotiation.cc:769] Sending connection context
0927 04:00:51.665502 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 04:00:51.665674 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":85,"threads_started":1}
I20260927 04:00:51.666435 29621 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.185.125" port: 35339 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 04:00:51 dist-test-slave-glgz krb5kdc[29390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.185.124@KRBTEST.COM
Sep 27 04:00:51 dist-test-slave-glgz krb5kdc[29390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481651, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.124@KRBTEST.COM for kudu/127.15.185.125@KRBTEST.COM
I20260927 04:00:51.672997 29625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:51.664310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34416 (local address 127.15.185.124:43821)
0927 04:00:51.664485 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 04:00:51.664489 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:00:51.664516 (+ 27us) server_negotiation.cc:407] Connection header received
0927 04:00:51.664569 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:00:51.664572 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:00:51.664585 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:00:51.664666 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:00:51.665285 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:51.666159 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:51.667069 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:51.667377 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:51.667462 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:51.670202 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:00:51.670211 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:00:51.670212 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:00:51.670219 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:00:51.671790 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:51.672134 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:51.672141 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:51.672141 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:51.672196 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:51.672389 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:51.672390 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:51.672391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:51.672536 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:00:51.672701 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 04:00:51.672747 (+ 46us) server_negotiation.cc:299] Negotiation successful
0927 04:00:51.672857 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":96,"threads_started":1}
I20260927 04:00:51.673745 29623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:51.666677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.125:35339 (local address 127.0.0.1:45476)
0927 04:00:51.666742 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:51.666753 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 04:00:51.666844 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:00:51.666980 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:00:51.666981 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:00:51.667148 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:00:51.667200 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:51.667201 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:51.667881 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:51.667882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:51.668775 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:51.668778 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:51.669220 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:51.669221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:51.669273 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:51.669656 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:00:51.669660 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:00:51.671120 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:00:51.672580 (+ 1460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:00:51.672581 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:00:51.672581 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:00:51.672890 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:00:51.673081 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:00:51.673082 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:00:51.673082 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:00:51.673142 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:00:51.673414 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:00:51.673415 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:00:51.673449 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 04:00:51.673549 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 04:00:51.673637 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 04:00:51.673794 29550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:51.666747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45476 (local address 127.15.185.125:35339)
0927 04:00:51.666810 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 04:00:51.666811 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 04:00:51.666819 (+ 8us) server_negotiation.cc:407] Connection header received
0927 04:00:51.666910 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:00:51.666911 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:00:51.666920 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:00:51.666976 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:00:51.667277 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:51.667793 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:51.668866 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:51.669128 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:51.669218 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:51.671191 (+ 1973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:00:51.671193 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:00:51.671193 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:00:51.671201 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:00:51.672511 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:51.672957 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:51.672958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:51.672959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:51.673015 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:51.673210 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:51.673211 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:51.673212 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:51.673355 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:00:51.673422 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 04:00:51.673579 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 04:00:51.673674 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 04:00:51.674525 29621 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.185.124" port: 43821 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 04:00:51 dist-test-slave-glgz krb5kdc[29390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
Sep 27 04:00:51 dist-test-slave-glgz krb5kdc[29390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481651, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.124@KRBTEST.COM for kudu/127.15.185.124@KRBTEST.COM
I20260927 04:00:51.681141 29623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:51.674788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.124:43821 (local address 127.0.0.1:34420)
0927 04:00:51.674850 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:51.674859 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 04:00:51.674899 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:00:51.675016 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:00:51.675016 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:00:51.675168 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:00:51.675197 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:51.675198 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:51.675811 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:51.675812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:51.676325 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:51.676327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:51.676775 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:51.676776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:51.676815 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:51.677185 (+ 370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:00:51.677190 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:00:51.678685 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:00:51.680214 (+ 1529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:00:51.680215 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:00:51.680215 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:00:51.680376 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:00:51.680549 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:00:51.680549 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:00:51.680550 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:00:51.680608 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:00:51.680840 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:00:51.680841 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:00:51.680877 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 04:00:51.680941 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 04:00:51.681033 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 04:00:51.681183 29625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:51.674823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34420 (local address 127.15.185.124:43821)
0927 04:00:51.674853 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0927 04:00:51.674855 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:00:51.674886 (+ 31us) server_negotiation.cc:407] Connection header received
0927 04:00:51.674951 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:00:51.674951 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:00:51.674960 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:00:51.675013 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:00:51.675264 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:51.675726 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:51.676403 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:51.676699 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:51.676776 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:51.678774 (+ 1998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:00:51.678775 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:00:51.678775 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:00:51.678781 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:00:51.680154 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:51.680431 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:51.680432 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:51.680432 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:51.680476 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:51.680666 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:51.680667 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:51.680667 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:51.680776 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:00:51.680847 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 04:00:51.680965 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 04:00:51.681079 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":14}
I20260927 04:00:51.681469 29475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:51.673858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42814 (local address 127.15.185.126:45857)
0927 04:00:51.673905 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 04:00:51.673907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:00:51.673917 (+ 10us) server_negotiation.cc:407] Connection header received
0927 04:00:51.673945 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:00:51.673946 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:00:51.673958 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:00:51.674016 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:00:51.674513 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:51.675026 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:51.675733 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:51.675934 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:51.676015 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:51.678418 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:00:51.678420 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:00:51.678421 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:00:51.678428 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:00:51.680088 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:51.680673 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:51.680674 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:51.680675 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:51.680734 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:51.680922 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:51.680923 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:51.680924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:51.681034 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:00:51.681243 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 04:00:51.681283 (+ 40us) server_negotiation.cc:299] Negotiation successful
0927 04:00:51.681357 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 04:00:51.683073 29621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe29f070521340928ed3b785ca16c139: Bootstrap starting.
I20260927 04:00:51.684012 29621 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe29f070521340928ed3b785ca16c139: Neither blocks nor log segments found. Creating new log.
I20260927 04:00:51.684324 29621 log.cc:824] T 00000000000000000000000000000000 P fe29f070521340928ed3b785ca16c139: Log is configured to *not* fsync() on all Append() calls
I20260927 04:00:51.685119 29621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe29f070521340928ed3b785ca16c139: No bootstrap required, opened a new log
Sep 27 04:00:51 dist-test-slave-glgz krb5kdc[29390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.185.125@KRBTEST.COM
I20260927 04:00:51.686678 29621 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe29f070521340928ed3b785ca16c139 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6cb4bf4a3554a9ab3a78dfe25853f0d" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45857 } } peers { permanent_uuid: "98fa76cd38b24a048dd743e2b43cef62" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 35339 } } peers { permanent_uuid: "fe29f070521340928ed3b785ca16c139" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 43821 } }
I20260927 04:00:51.686854 29621 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe29f070521340928ed3b785ca16c139 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:00:51.686897 29621 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe29f070521340928ed3b785ca16c139 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe29f070521340928ed3b785ca16c139, State: Initialized, Role: FOLLOWER
I20260927 04:00:51.687032 29621 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe29f070521340928ed3b785ca16c139 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6cb4bf4a3554a9ab3a78dfe25853f0d" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45857 } } peers { permanent_uuid: "98fa76cd38b24a048dd743e2b43cef62" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 35339 } } peers { permanent_uuid: "fe29f070521340928ed3b785ca16c139" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 43821 } }
I20260927 04:00:51.687348 29627 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe29f070521340928ed3b785ca16c139 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6cb4bf4a3554a9ab3a78dfe25853f0d" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45857 } } peers { permanent_uuid: "98fa76cd38b24a048dd743e2b43cef62" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 35339 } } peers { permanent_uuid: "fe29f070521340928ed3b785ca16c139" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 43821 } } }
I20260927 04:00:51.687457 29627 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe29f070521340928ed3b785ca16c139 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 04:00:51.687402 29621 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe29f070521340928ed3b785ca16c139 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:00:51.689615 29550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:51.682090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45486 (local address 127.15.185.125:35339)
0927 04:00:51.682135 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 04:00:51.682137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:00:51.682183 (+ 46us) server_negotiation.cc:407] Connection header received
0927 04:00:51.682231 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:00:51.682232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:00:51.682242 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:00:51.682304 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:00:51.682760 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:51.683461 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:51.684120 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:51.684405 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:51.684486 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:51.686666 (+ 2180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:00:51.686668 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:00:51.686668 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:00:51.686675 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:00:51.688443 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:51.688757 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:51.688758 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:51.688758 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:51.688823 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:51.689053 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:51.689054 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:51.689055 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:51.689218 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:00:51.689406 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 04:00:51.689450 (+ 44us) server_negotiation.cc:299] Negotiation successful
0927 04:00:51.689522 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260927 04:00:51.689787 29638 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fe29f070521340928ed3b785ca16c139: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 04:00:51.689862 29638 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 04:00:51.691620 16101 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 04:00:51.690297 29553 master_runner.cc:186] Error getting master registration for 127.15.185.126:45857: OK, instance_id { permanent_uuid: "f6cb4bf4a3554a9ab3a78dfe25853f0d" instance_seqno: 1790481651356928 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 04:00:51.693463 29553 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 04:00:51.698936 29471 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.185.124" port: 43821 } attempt: 2
I20260927 04:00:51.702107 29550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:51.693104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45500 (local address 127.15.185.125:35339)
0927 04:00:51.693146 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 04:00:51.693148 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:00:51.693720 (+ 572us) server_negotiation.cc:407] Connection header received
0927 04:00:51.693795 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:00:51.693796 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:00:51.693806 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:00:51.693855 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:00:51.694455 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:51.695194 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:51.696286 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:51.696577 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:51.696617 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:51.698462 (+ 1845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:00:51.698464 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:00:51.698464 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:00:51.698471 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:00:51.700867 (+ 2396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:51.701230 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:51.701232 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:51.701232 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:51.701291 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:51.701493 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:51.701494 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:51.701495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:51.701627 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:00:51.701704 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 04:00:51.701935 (+ 231us) server_negotiation.cc:299] Negotiation successful
0927 04:00:51.702017 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 04:00:51.702725 29625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:51.693472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34426 (local address 127.15.185.124:43821)
0927 04:00:51.693516 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 04:00:51.693519 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:00:51.693875 (+ 356us) server_negotiation.cc:407] Connection header received
0927 04:00:51.693962 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:00:51.693963 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:00:51.693972 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:00:51.694001 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:00:51.694732 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:51.695495 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:51.696501 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:51.696784 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:51.696825 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:51.698469 (+ 1644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:00:51.698470 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:00:51.698470 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:00:51.698477 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:00:51.700269 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:51.701482 (+ 1213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:51.701484 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:51.701484 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:51.701541 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:51.701738 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:51.701739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:51.701739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:51.701897 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:00:51.701989 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 04:00:51.702547 (+ 558us) server_negotiation.cc:299] Negotiation successful
0927 04:00:51.702630 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 04:00:51.702914 29475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:51.693243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42816 (local address 127.15.185.126:45857)
0927 04:00:51.693280 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 04:00:51.693282 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:00:51.693399 (+ 117us) server_negotiation.cc:407] Connection header received
0927 04:00:51.693629 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:00:51.693630 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:00:51.693653 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:00:51.693720 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:00:51.694433 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:51.695201 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:51.696287 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:51.696577 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:51.696617 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:51.698510 (+ 1893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:00:51.698511 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:00:51.698512 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:00:51.698518 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:00:51.701380 (+ 2862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:51.701749 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:51.701750 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:51.701750 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:51.701804 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:51.702008 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:51.702008 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:51.702009 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:51.702130 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:00:51.702195 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 04:00:51.702755 (+ 560us) server_negotiation.cc:299] Negotiation successful
0927 04:00:51.702837 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
Sep 27 04:00:51 dist-test-slave-glgz krb5kdc[29390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481651, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.126@KRBTEST.COM for kudu/127.15.185.124@KRBTEST.COM
I20260927 04:00:51.709811 29474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:51.699277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.124:43821 (local address 127.0.0.1:34436)
0927 04:00:51.699372 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:51.699382 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 04:00:51.699417 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:00:51.699779 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:00:51.699780 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:00:51.699950 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:00:51.699993 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:51.699994 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:51.703913 (+ 3919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:51.703914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:51.704384 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:51.704386 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:51.704842 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:51.704843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:51.704883 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:51.705297 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:00:51.705302 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:00:51.706620 (+ 1318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:00:51.708563 (+ 1943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:00:51.708565 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:00:51.708565 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:00:51.708759 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:00:51.708980 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:00:51.708981 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:00:51.708981 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:00:51.709059 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:00:51.709522 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:00:51.709523 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:00:51.709576 (+ 53us) client_negotiation.cc:769] Sending connection context
0927 04:00:51.709614 (+ 38us) client_negotiation.cc:240] Negotiation successful
0927 04:00:51.709737 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 04:00:51.710202 29647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:51.699495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34436 (local address 127.15.185.124:43821)
0927 04:00:51.699644 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 04:00:51.699646 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:00:51.699653 (+ 7us) server_negotiation.cc:407] Connection header received
0927 04:00:51.699672 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:00:51.699672 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:00:51.699689 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:00:51.700255 (+ 566us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:00:51.700298 (+ 43us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:51.703843 (+ 3545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:51.704460 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:51.704768 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:51.704839 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:51.706704 (+ 1865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:00:51.706705 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:00:51.706706 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:00:51.706712 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:00:51.708476 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:51.708833 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:51.708834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:51.708834 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:51.708893 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:51.709290 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:51.709291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:51.709292 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:51.709442 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:00:51.709961 (+ 519us) server_negotiation.cc:1035] Waiting for connection context
0927 04:00:51.710010 (+ 49us) server_negotiation.cc:299] Negotiation successful
0927 04:00:51.710087 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":100,"threads_started":1}
I20260927 04:00:51.711856 29471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6cb4bf4a3554a9ab3a78dfe25853f0d: Bootstrap starting.
I20260927 04:00:51.712611 29471 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6cb4bf4a3554a9ab3a78dfe25853f0d: Neither blocks nor log segments found. Creating new log.
I20260927 04:00:51.712913 29471 log.cc:824] T 00000000000000000000000000000000 P f6cb4bf4a3554a9ab3a78dfe25853f0d: Log is configured to *not* fsync() on all Append() calls
I20260927 04:00:51.713793 29471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6cb4bf4a3554a9ab3a78dfe25853f0d: No bootstrap required, opened a new log
I20260927 04:00:51.716135 29471 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6cb4bf4a3554a9ab3a78dfe25853f0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6cb4bf4a3554a9ab3a78dfe25853f0d" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45857 } } peers { permanent_uuid: "98fa76cd38b24a048dd743e2b43cef62" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 35339 } } peers { permanent_uuid: "fe29f070521340928ed3b785ca16c139" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 43821 } }
I20260927 04:00:51.716272 29471 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6cb4bf4a3554a9ab3a78dfe25853f0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:00:51.716306 29471 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6cb4bf4a3554a9ab3a78dfe25853f0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6cb4bf4a3554a9ab3a78dfe25853f0d, State: Initialized, Role: FOLLOWER
I20260927 04:00:51.716444 29471 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6cb4bf4a3554a9ab3a78dfe25853f0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6cb4bf4a3554a9ab3a78dfe25853f0d" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45857 } } peers { permanent_uuid: "98fa76cd38b24a048dd743e2b43cef62" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 35339 } } peers { permanent_uuid: "fe29f070521340928ed3b785ca16c139" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 43821 } }
I20260927 04:00:51.716713 29649 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6cb4bf4a3554a9ab3a78dfe25853f0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6cb4bf4a3554a9ab3a78dfe25853f0d" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45857 } } peers { permanent_uuid: "98fa76cd38b24a048dd743e2b43cef62" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 35339 } } peers { permanent_uuid: "fe29f070521340928ed3b785ca16c139" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 43821 } } }
I20260927 04:00:51.716753 29471 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6cb4bf4a3554a9ab3a78dfe25853f0d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:00:51.716817 29649 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6cb4bf4a3554a9ab3a78dfe25853f0d [sys.catalog]: This master's current role is: FOLLOWER
W20260927 04:00:51.718763 29660 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f6cb4bf4a3554a9ab3a78dfe25853f0d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 04:00:51.719028 29660 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 04:00:51.719748 29403 master_runner.cc:186] Error getting master registration for 127.15.185.125:35339: OK, instance_id { permanent_uuid: "98fa76cd38b24a048dd743e2b43cef62" instance_seqno: 1790481651504193 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 04:00:51.721971 29403 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 04:00:51.739415 29546 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.185.124" port: 43821 } attempt: 3
Sep 27 04:00:51 dist-test-slave-glgz krb5kdc[29390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481651, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.125@KRBTEST.COM for kudu/127.15.185.124@KRBTEST.COM
I20260927 04:00:51.747274 29647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:51.739863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34442 (local address 127.15.185.124:43821)
0927 04:00:51.739905 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 04:00:51.739907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:00:51.739938 (+ 31us) server_negotiation.cc:407] Connection header received
0927 04:00:51.740078 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:00:51.740079 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:00:51.740102 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:00:51.740161 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:00:51.740590 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:51.741141 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:51.741970 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:51.742198 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:51.742277 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:51.744266 (+ 1989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:00:51.744268 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:00:51.744268 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:00:51.744276 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:00:51.745837 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:51.746217 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:51.746218 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:51.746219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:51.746293 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:51.746709 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:51.746710 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:51.746710 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:51.746842 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:00:51.746918 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 04:00:51.747105 (+ 187us) server_negotiation.cc:299] Negotiation successful
0927 04:00:51.747182 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 04:00:51.747285 29548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:51.739820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.124:43821 (local address 127.0.0.1:34442)
0927 04:00:51.739881 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:51.739895 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 04:00:51.739955 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:00:51.740167 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:00:51.740167 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:00:51.740431 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:00:51.740517 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:51.740520 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:51.741223 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:51.741224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:51.741881 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:51.741883 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:51.742272 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:51.742273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:51.742326 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:51.742805 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:00:51.742810 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:00:51.744177 (+ 1367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:00:51.745920 (+ 1743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:00:51.745921 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:00:51.745922 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:00:51.746142 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:00:51.746559 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:00:51.746560 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:00:51.746560 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:00:51.746646 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:00:51.746910 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:00:51.746911 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:00:51.746957 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 04:00:51.747037 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 04:00:51.747175 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 04:00:51.748739 29546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98fa76cd38b24a048dd743e2b43cef62: Bootstrap starting.
I20260927 04:00:51.749392 29546 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98fa76cd38b24a048dd743e2b43cef62: Neither blocks nor log segments found. Creating new log.
I20260927 04:00:51.749687 29546 log.cc:824] T 00000000000000000000000000000000 P 98fa76cd38b24a048dd743e2b43cef62: Log is configured to *not* fsync() on all Append() calls
I20260927 04:00:51.750442 29546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98fa76cd38b24a048dd743e2b43cef62: No bootstrap required, opened a new log
I20260927 04:00:51.751916 29546 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98fa76cd38b24a048dd743e2b43cef62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6cb4bf4a3554a9ab3a78dfe25853f0d" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45857 } } peers { permanent_uuid: "98fa76cd38b24a048dd743e2b43cef62" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 35339 } } peers { permanent_uuid: "fe29f070521340928ed3b785ca16c139" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 43821 } }
I20260927 04:00:51.752063 29546 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98fa76cd38b24a048dd743e2b43cef62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:00:51.752140 29546 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98fa76cd38b24a048dd743e2b43cef62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98fa76cd38b24a048dd743e2b43cef62, State: Initialized, Role: FOLLOWER
I20260927 04:00:51.752290 29546 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98fa76cd38b24a048dd743e2b43cef62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6cb4bf4a3554a9ab3a78dfe25853f0d" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45857 } } peers { permanent_uuid: "98fa76cd38b24a048dd743e2b43cef62" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 35339 } } peers { permanent_uuid: "fe29f070521340928ed3b785ca16c139" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 43821 } }
I20260927 04:00:51.752552 29663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98fa76cd38b24a048dd743e2b43cef62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6cb4bf4a3554a9ab3a78dfe25853f0d" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45857 } } peers { permanent_uuid: "98fa76cd38b24a048dd743e2b43cef62" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 35339 } } peers { permanent_uuid: "fe29f070521340928ed3b785ca16c139" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 43821 } } }
I20260927 04:00:51.752612 29546 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98fa76cd38b24a048dd743e2b43cef62 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:00:51.752681 29663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98fa76cd38b24a048dd743e2b43cef62 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 04:00:51.754443 29674 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 98fa76cd38b24a048dd743e2b43cef62: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 04:00:51.754544 29674 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 04:00:51.758584 29478 master_runner.cc:263] No leader master found from master 98fa76cd38b24a048dd743e2b43cef62
I20260927 04:00:51.758658 29478 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 04:00:51.854996 29424 tablet.cc:2369] T 00000000000000000000000000000000 P f6cb4bf4a3554a9ab3a78dfe25853f0d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 04:00:51.900544 29574 tablet.cc:2369] T 00000000000000000000000000000000 P fe29f070521340928ed3b785ca16c139: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 04:00:51.956233 29649 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6cb4bf4a3554a9ab3a78dfe25853f0d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:00:51.956359 29649 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6cb4bf4a3554a9ab3a78dfe25853f0d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6cb4bf4a3554a9ab3a78dfe25853f0d" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45857 } } peers { permanent_uuid: "98fa76cd38b24a048dd743e2b43cef62" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 35339 } } peers { permanent_uuid: "fe29f070521340928ed3b785ca16c139" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 43821 } }
I20260927 04:00:51.956653 29649 leader_election.cc:288] T 00000000000000000000000000000000 P f6cb4bf4a3554a9ab3a78dfe25853f0d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98fa76cd38b24a048dd743e2b43cef62 (127.15.185.125:35339), fe29f070521340928ed3b785ca16c139 (127.15.185.124:43821)
I20260927 04:00:51.957007 29595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f6cb4bf4a3554a9ab3a78dfe25853f0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe29f070521340928ed3b785ca16c139" is_pre_election: true
I20260927 04:00:51.957011 29520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f6cb4bf4a3554a9ab3a78dfe25853f0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98fa76cd38b24a048dd743e2b43cef62" is_pre_election: true
I20260927 04:00:51.957175 29595 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fe29f070521340928ed3b785ca16c139 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6cb4bf4a3554a9ab3a78dfe25853f0d in term 0.
I20260927 04:00:51.957173 29520 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 98fa76cd38b24a048dd743e2b43cef62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6cb4bf4a3554a9ab3a78dfe25853f0d in term 0.
I20260927 04:00:51.957407 29419 leader_election.cc:302] T 00000000000000000000000000000000 P f6cb4bf4a3554a9ab3a78dfe25853f0d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f6cb4bf4a3554a9ab3a78dfe25853f0d, fe29f070521340928ed3b785ca16c139; no voters:
I20260927 04:00:51.957558 29649 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6cb4bf4a3554a9ab3a78dfe25853f0d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 04:00:51.957619 29649 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6cb4bf4a3554a9ab3a78dfe25853f0d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 04:00:51.957646 29649 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6cb4bf4a3554a9ab3a78dfe25853f0d [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:00:51.958364 29649 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6cb4bf4a3554a9ab3a78dfe25853f0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6cb4bf4a3554a9ab3a78dfe25853f0d" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45857 } } peers { permanent_uuid: "98fa76cd38b24a048dd743e2b43cef62" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 35339 } } peers { permanent_uuid: "fe29f070521340928ed3b785ca16c139" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 43821 } }
I20260927 04:00:51.958501 29649 leader_election.cc:288] T 00000000000000000000000000000000 P f6cb4bf4a3554a9ab3a78dfe25853f0d [CANDIDATE]: Term 1 election: Requested vote from peers 98fa76cd38b24a048dd743e2b43cef62 (127.15.185.125:35339), fe29f070521340928ed3b785ca16c139 (127.15.185.124:43821)
I20260927 04:00:51.958698 29520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f6cb4bf4a3554a9ab3a78dfe25853f0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98fa76cd38b24a048dd743e2b43cef62"
I20260927 04:00:51.958720 29595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f6cb4bf4a3554a9ab3a78dfe25853f0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe29f070521340928ed3b785ca16c139"
I20260927 04:00:51.958801 29595 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe29f070521340928ed3b785ca16c139 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:00:51.958788 29520 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98fa76cd38b24a048dd743e2b43cef62 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:00:51.959704 29595 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fe29f070521340928ed3b785ca16c139 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6cb4bf4a3554a9ab3a78dfe25853f0d in term 1.
I20260927 04:00:51.959705 29520 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 98fa76cd38b24a048dd743e2b43cef62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6cb4bf4a3554a9ab3a78dfe25853f0d in term 1.
I20260927 04:00:51.959885 29419 leader_election.cc:302] T 00000000000000000000000000000000 P f6cb4bf4a3554a9ab3a78dfe25853f0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f6cb4bf4a3554a9ab3a78dfe25853f0d, fe29f070521340928ed3b785ca16c139; no voters:
I20260927 04:00:51.959986 29649 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6cb4bf4a3554a9ab3a78dfe25853f0d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:00:51.960119 29649 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6cb4bf4a3554a9ab3a78dfe25853f0d [term 1 LEADER]: Becoming Leader. State: Replica: f6cb4bf4a3554a9ab3a78dfe25853f0d, State: Running, Role: LEADER
I20260927 04:00:51.960228 29649 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6cb4bf4a3554a9ab3a78dfe25853f0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6cb4bf4a3554a9ab3a78dfe25853f0d" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45857 } } peers { permanent_uuid: "98fa76cd38b24a048dd743e2b43cef62" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 35339 } } peers { permanent_uuid: "fe29f070521340928ed3b785ca16c139" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 43821 } }
I20260927 04:00:51.960700 29676 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6cb4bf4a3554a9ab3a78dfe25853f0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6cb4bf4a3554a9ab3a78dfe25853f0d. Latest consensus state: current_term: 1 leader_uuid: "f6cb4bf4a3554a9ab3a78dfe25853f0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6cb4bf4a3554a9ab3a78dfe25853f0d" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45857 } } peers { permanent_uuid: "98fa76cd38b24a048dd743e2b43cef62" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 35339 } } peers { permanent_uuid: "fe29f070521340928ed3b785ca16c139" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 43821 } } }
I20260927 04:00:51.960793 29676 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6cb4bf4a3554a9ab3a78dfe25853f0d [sys.catalog]: This master's current role is: LEADER
I20260927 04:00:51.961004 29678 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:00:51.961198 29678 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:00:51.962529 29520 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 98fa76cd38b24a048dd743e2b43cef62 [term 1 FOLLOWER]: Refusing update from remote peer f6cb4bf4a3554a9ab3a78dfe25853f0d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:00:51.962576 29595 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fe29f070521340928ed3b785ca16c139 [term 1 FOLLOWER]: Refusing update from remote peer f6cb4bf4a3554a9ab3a78dfe25853f0d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:00:51.962879 29649 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f6cb4bf4a3554a9ab3a78dfe25853f0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe29f070521340928ed3b785ca16c139" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 43821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:00:51.962978 29676 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f6cb4bf4a3554a9ab3a78dfe25853f0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "98fa76cd38b24a048dd743e2b43cef62" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 35339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:00:51.964186 29627 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe29f070521340928ed3b785ca16c139 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6cb4bf4a3554a9ab3a78dfe25853f0d. Latest consensus state: current_term: 1 leader_uuid: "f6cb4bf4a3554a9ab3a78dfe25853f0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6cb4bf4a3554a9ab3a78dfe25853f0d" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45857 } } peers { permanent_uuid: "98fa76cd38b24a048dd743e2b43cef62" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 35339 } } peers { permanent_uuid: "fe29f070521340928ed3b785ca16c139" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 43821 } } }
I20260927 04:00:51.964185 29663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98fa76cd38b24a048dd743e2b43cef62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6cb4bf4a3554a9ab3a78dfe25853f0d. Latest consensus state: current_term: 1 leader_uuid: "f6cb4bf4a3554a9ab3a78dfe25853f0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6cb4bf4a3554a9ab3a78dfe25853f0d" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45857 } } peers { permanent_uuid: "98fa76cd38b24a048dd743e2b43cef62" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 35339 } } peers { permanent_uuid: "fe29f070521340928ed3b785ca16c139" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 43821 } } }
I20260927 04:00:51.964273 29663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98fa76cd38b24a048dd743e2b43cef62 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 04:00:51.964273 29627 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe29f070521340928ed3b785ca16c139 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 04:00:51.965139 29649 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6cb4bf4a3554a9ab3a78dfe25853f0d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f6cb4bf4a3554a9ab3a78dfe25853f0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6cb4bf4a3554a9ab3a78dfe25853f0d" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45857 } } peers { permanent_uuid: "98fa76cd38b24a048dd743e2b43cef62" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 35339 } } peers { permanent_uuid: "fe29f070521340928ed3b785ca16c139" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 43821 } } }
I20260927 04:00:51.965243 29649 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6cb4bf4a3554a9ab3a78dfe25853f0d [sys.catalog]: This master's current role is: LEADER
I20260927 04:00:51.965287 29649 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6cb4bf4a3554a9ab3a78dfe25853f0d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f6cb4bf4a3554a9ab3a78dfe25853f0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6cb4bf4a3554a9ab3a78dfe25853f0d" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45857 } } peers { permanent_uuid: "98fa76cd38b24a048dd743e2b43cef62" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 35339 } } peers { permanent_uuid: "fe29f070521340928ed3b785ca16c139" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 43821 } } }
I20260927 04:00:51.965328 29649 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6cb4bf4a3554a9ab3a78dfe25853f0d [sys.catalog]: This master's current role is: LEADER
I20260927 04:00:51.965746 29627 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe29f070521340928ed3b785ca16c139 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f6cb4bf4a3554a9ab3a78dfe25853f0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6cb4bf4a3554a9ab3a78dfe25853f0d" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45857 } } peers { permanent_uuid: "98fa76cd38b24a048dd743e2b43cef62" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 35339 } } peers { permanent_uuid: "fe29f070521340928ed3b785ca16c139" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 43821 } } }
I20260927 04:00:51.965839 29627 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe29f070521340928ed3b785ca16c139 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 04:00:51.965867 29678 catalog_manager.cc:1347] Generated new cluster ID: 4d7c83a89b0445cba8a25371f2453817
I20260927 04:00:51.965919 29678 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:00:51.965947 29663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98fa76cd38b24a048dd743e2b43cef62 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f6cb4bf4a3554a9ab3a78dfe25853f0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6cb4bf4a3554a9ab3a78dfe25853f0d" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45857 } } peers { permanent_uuid: "98fa76cd38b24a048dd743e2b43cef62" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 35339 } } peers { permanent_uuid: "fe29f070521340928ed3b785ca16c139" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 43821 } } }
I20260927 04:00:51.966035 29663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98fa76cd38b24a048dd743e2b43cef62 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 04:00:51.966215 29680 mvcc.cc:205] Tried to move back new op lower bound from 7333812846436306944 to 7333812846429818880. Current Snapshot: MvccSnapshot[applied={T|T < 7333812846436306944}]
I20260927 04:00:51.979889 29678 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:00:51.980504 29678 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:00:51.993212 29678 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6cb4bf4a3554a9ab3a78dfe25853f0d: Generated new TSK 0
I20260927 04:00:51.993500 29678 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 04:00:52.690327 29638 catalog_manager.cc:1259] Loaded cluster ID: 4d7c83a89b0445cba8a25371f2453817
I20260927 04:00:52.690407 29638 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fe29f070521340928ed3b785ca16c139: loading cluster ID for follower catalog manager: success
I20260927 04:00:52.691344 29638 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fe29f070521340928ed3b785ca16c139: acquiring CA information for follower catalog manager: success
I20260927 04:00:52.691679 29638 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fe29f070521340928ed3b785ca16c139: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 04:00:52.755018 29674 catalog_manager.cc:1259] Loaded cluster ID: 4d7c83a89b0445cba8a25371f2453817
I20260927 04:00:52.755133 29674 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 98fa76cd38b24a048dd743e2b43cef62: loading cluster ID for follower catalog manager: success
I20260927 04:00:52.755937 29674 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 98fa76cd38b24a048dd743e2b43cef62: acquiring CA information for follower catalog manager: success
I20260927 04:00:52.756253 29674 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 98fa76cd38b24a048dd743e2b43cef62: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 04:00:53.064217 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 29403
I20260927 04:00:53.069654 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 29478
I20260927 04:00:53.074419 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 29553
2026-09-27T04:00:53Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3001 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481498018407-16101-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:00:53 dist-test-slave-glgz krb5kdc[29693](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:00:53 dist-test-slave-glgz krb5kdc[29693](info): set up 2 sockets
Sep 27 04:00:53 dist-test-slave-glgz krb5kdc[29693](info): commencing operation
krb5kdc: starting...
W20260927 04:00:54.123657 16101 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Sep 27 04:00:54 dist-test-slave-glgz krb5kdc[29693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:00:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:00:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.185.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab.
Entry for principal kudu/127.15.185.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab.
I20260927 04:00:54.242592 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481498018407-16101-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481498018407-16101-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481498018407-16101-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:33159
--webserver_interface=127.15.185.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:46837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.185.126:33159
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:00:54.324222 29706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:00:54.324462 29706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:00:54.324501 29706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:00:54.325666 29706 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:00:54.325736 29706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:00:54.325762 29706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:00:54.325791 29706 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:00:54.325814 29706 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:00:54.327384 29706 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:46837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481498018407-16101-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481498018407-16101-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.185.126:33159
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:33159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481498018407-16101-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:00:54.327651 29706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:00:54.327874 29706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 04:00:54.332805 29706 server_base.cc:1034] running on GCE node
W20260927 04:00:54.332768 29712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:00:54.332775 29711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:00:54.332784 29714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:00:54.333194 29706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:00:54.333508 29706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:00:54.334684 29706 hybrid_clock.cc:648] HybridClock initialized: now 1790481654334677 us; error 38 us; skew 500 ppm
Sep 27 04:00:54 dist-test-slave-glgz krb5kdc[29693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481654, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:00:54.337180 29706 init.cc:377] Logged in from keytab as kudu/127.15.185.126@KRBTEST.COM (short username kudu)
I20260927 04:00:54.337621 29706 webserver.cc:460] Webserver started at http://127.15.185.126:38457/ using document root <none> and password file <none>
I20260927 04:00:54.338057 29706 fs_manager.cc:359] Metadata directory not provided
I20260927 04:00:54.338143 29706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:00:54.338251 29706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:00:54.339427 29706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481498018407-16101-0/minicluster-data/master-0/data/instance:
uuid: "d6ec67a499b34c0292dff8d9051bedea"
format_stamp: "Formatted at 2026-09-27 04:00:54 on dist-test-slave-glgz"
server_key: "3dca5170c9be7392096db5ab11cf32ff"
server_key_iv: "4db81d8b42f41913fb302bb4a0365a5b"
server_key_version: "encryptionkey@0"
I20260927 04:00:54.339751 29706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481498018407-16101-0/minicluster-data/master-0/wal/instance:
uuid: "d6ec67a499b34c0292dff8d9051bedea"
format_stamp: "Formatted at 2026-09-27 04:00:54 on dist-test-slave-glgz"
server_key: "3dca5170c9be7392096db5ab11cf32ff"
server_key_iv: "4db81d8b42f41913fb302bb4a0365a5b"
server_key_version: "encryptionkey@0"
I20260927 04:00:54.341038 29706 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:00:54.341676 29721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:00:54.341863 29706 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 04:00:54.341931 29706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481498018407-16101-0/minicluster-data/master-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481498018407-16101-0/minicluster-data/master-0/wal
uuid: "d6ec67a499b34c0292dff8d9051bedea"
format_stamp: "Formatted at 2026-09-27 04:00:54 on dist-test-slave-glgz"
server_key: "3dca5170c9be7392096db5ab11cf32ff"
server_key_iv: "4db81d8b42f41913fb302bb4a0365a5b"
server_key_version: "encryptionkey@0"
I20260927 04:00:54.342031 29706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481498018407-16101-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481498018407-16101-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481498018407-16101-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:00:54.358251 29706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:00:54.360191 29706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:00:54.360348 29706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:00:54.364337 29706 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.126:33159
I20260927 04:00:54.364400 29773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.126:33159 every 8 connection(s)
I20260927 04:00:54.364766 29706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
I20260927 04:00:54.365603 29774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:00:54.367287 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 29706
I20260927 04:00:54.367414 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481498018407-16101-0/minicluster-data/master-0/wal/instance
I20260927 04:00:54.367632 16101 external_mini_cluster.cc:1428] Setting key 17e07b5ae39459b823479f813be518d5
I20260927 04:00:54.368284 29774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6ec67a499b34c0292dff8d9051bedea: Bootstrap starting.
I20260927 04:00:54.368996 29774 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6ec67a499b34c0292dff8d9051bedea: Neither blocks nor log segments found. Creating new log.
I20260927 04:00:54.369324 29774 log.cc:824] T 00000000000000000000000000000000 P d6ec67a499b34c0292dff8d9051bedea: Log is configured to *not* fsync() on all Append() calls
I20260927 04:00:54.370142 29774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6ec67a499b34c0292dff8d9051bedea: No bootstrap required, opened a new log
I20260927 04:00:54.372500 29774 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6ec67a499b34c0292dff8d9051bedea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6ec67a499b34c0292dff8d9051bedea" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 33159 } }
I20260927 04:00:54.372670 29774 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6ec67a499b34c0292dff8d9051bedea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:00:54.372705 29774 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6ec67a499b34c0292dff8d9051bedea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6ec67a499b34c0292dff8d9051bedea, State: Initialized, Role: FOLLOWER
I20260927 04:00:54.372835 29774 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6ec67a499b34c0292dff8d9051bedea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6ec67a499b34c0292dff8d9051bedea" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 33159 } }
I20260927 04:00:54.372933 29774 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6ec67a499b34c0292dff8d9051bedea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:00:54.372967 29774 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6ec67a499b34c0292dff8d9051bedea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:00:54.373059 29774 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6ec67a499b34c0292dff8d9051bedea [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:00:54.373677 29774 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6ec67a499b34c0292dff8d9051bedea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6ec67a499b34c0292dff8d9051bedea" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 33159 } }
I20260927 04:00:54.373764 29774 leader_election.cc:302] T 00000000000000000000000000000000 P d6ec67a499b34c0292dff8d9051bedea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6ec67a499b34c0292dff8d9051bedea; no voters:
I20260927 04:00:54.373948 29774 leader_election.cc:288] T 00000000000000000000000000000000 P d6ec67a499b34c0292dff8d9051bedea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:00:54.373972 29779 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6ec67a499b34c0292dff8d9051bedea [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:00:54.374082 29779 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6ec67a499b34c0292dff8d9051bedea [term 1 LEADER]: Becoming Leader. State: Replica: d6ec67a499b34c0292dff8d9051bedea, State: Running, Role: LEADER
I20260927 04:00:54.374212 29774 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6ec67a499b34c0292dff8d9051bedea [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:00:54.374224 29779 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6ec67a499b34c0292dff8d9051bedea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6ec67a499b34c0292dff8d9051bedea" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 33159 } }
I20260927 04:00:54.374678 29779 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6ec67a499b34c0292dff8d9051bedea [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6ec67a499b34c0292dff8d9051bedea. Latest consensus state: current_term: 1 leader_uuid: "d6ec67a499b34c0292dff8d9051bedea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6ec67a499b34c0292dff8d9051bedea" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 33159 } } }
I20260927 04:00:54.374691 29781 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6ec67a499b34c0292dff8d9051bedea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6ec67a499b34c0292dff8d9051bedea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6ec67a499b34c0292dff8d9051bedea" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 33159 } } }
I20260927 04:00:54.374855 29779 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6ec67a499b34c0292dff8d9051bedea [sys.catalog]: This master's current role is: LEADER
I20260927 04:00:54.374867 29781 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6ec67a499b34c0292dff8d9051bedea [sys.catalog]: This master's current role is: LEADER
I20260927 04:00:54.375093 29788 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 27 04:00:54 dist-test-slave-glgz krb5kdc[29693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 04:00:54.375651 29788 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:00:54.378249 29788 catalog_manager.cc:1347] Generated new cluster ID: 5f8dc6aa96d748c085b3678b9c93174c
I20260927 04:00:54.378300 29788 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:00:54.379416 29777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:54.368132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39752 (local address 127.15.185.126:33159)
0927 04:00:54.368327 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 04:00:54.368337 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 04:00:54.368380 (+ 43us) server_negotiation.cc:407] Connection header received
0927 04:00:54.368972 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:00:54.368981 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:00:54.369046 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:00:54.369216 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:00:54.370003 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:54.371425 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:54.372193 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:54.372812 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:54.372923 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:54.375827 (+ 2904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:00:54.375832 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:00:54.375832 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:00:54.375846 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:00:54.377698 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:54.378119 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:54.378123 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:54.378123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:54.378234 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:54.378483 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:54.378485 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:54.378485 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:54.378770 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:00:54.378923 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 04:00:54.379088 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 04:00:54.379216 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":87,"threads_started":1}
I20260927 04:00:54.380615 16101 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 04:00:54.387247 29788 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:00:54.387938 29788 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:00:54.399245 29788 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6ec67a499b34c0292dff8d9051bedea: Generated new TSK 0
I20260927 04:00:54.399484 29788 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 04:00:54.434711 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 29706
2026-09-27T04:00:54Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1357 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:00:54 dist-test-slave-glgz krb5kdc[29801](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:00:54 dist-test-slave-glgz krb5kdc[29801](info): set up 2 sockets
Sep 27 04:00:54 dist-test-slave-glgz krb5kdc[29801](info): commencing operation
krb5kdc: starting...
W20260927 04:00:55.467475 16101 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Sep 27 04:00:55 dist-test-slave-glgz krb5kdc[29801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:00:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:00:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.185.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab.
Entry for principal kudu/127.15.185.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab.
I20260927 04:00:55.590262 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:45853
--webserver_interface=127.15.185.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:39093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.185.126:45853
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:00:55.671025 29814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:00:55.671260 29814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:00:55.671284 29814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:00:55.672449 29814 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:00:55.672492 29814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:00:55.672505 29814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:00:55.672518 29814 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:00:55.672530 29814 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:00:55.674041 29814 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:39093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.185.126:45853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:45853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:00:55.674306 29814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:00:55.674561 29814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:00:55.679425 29819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:00:55.679484 29820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:00:55.679474 29814 server_base.cc:1034] running on GCE node
W20260927 04:00:55.679425 29822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:00:55.679859 29814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:00:55.680150 29814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:00:55.681334 29814 hybrid_clock.cc:648] HybridClock initialized: now 1790481655681332 us; error 40 us; skew 500 ppm
Sep 27 04:00:55 dist-test-slave-glgz krb5kdc[29801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481655, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:00:55.683864 29814 init.cc:377] Logged in from keytab as kudu/127.15.185.126@KRBTEST.COM (short username kudu)
I20260927 04:00:55.684253 29814 webserver.cc:460] Webserver started at http://127.15.185.126:33989/ using document root <none> and password file <none>
I20260927 04:00:55.684435 29814 fs_manager.cc:359] Metadata directory not provided
I20260927 04:00:55.684477 29814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:00:55.684567 29814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:00:55.685580 29814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/master-0/data/instance:
uuid: "8c3e092adfae4d008e6c22c1c0c22a34"
format_stamp: "Formatted at 2026-09-27 04:00:55 on dist-test-slave-glgz"
server_key: "828e675d78684da1271578c26259e0d5"
server_key_iv: "6c8a661f1af8a136fb39b7dc3eae775d"
server_key_version: "encryptionkey@0"
I20260927 04:00:55.685881 29814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/master-0/wal/instance:
uuid: "8c3e092adfae4d008e6c22c1c0c22a34"
format_stamp: "Formatted at 2026-09-27 04:00:55 on dist-test-slave-glgz"
server_key: "828e675d78684da1271578c26259e0d5"
server_key_iv: "6c8a661f1af8a136fb39b7dc3eae775d"
server_key_version: "encryptionkey@0"
I20260927 04:00:55.687186 29814 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 04:00:55.687963 29829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:00:55.688139 29814 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:00:55.688253 29814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/master-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/master-0/wal
uuid: "8c3e092adfae4d008e6c22c1c0c22a34"
format_stamp: "Formatted at 2026-09-27 04:00:55 on dist-test-slave-glgz"
server_key: "828e675d78684da1271578c26259e0d5"
server_key_iv: "6c8a661f1af8a136fb39b7dc3eae775d"
server_key_version: "encryptionkey@0"
I20260927 04:00:55.688334 29814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:00:55.698119 29814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:00:55.700482 29814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:00:55.700642 29814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:00:55.704641 29814 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.126:45853
I20260927 04:00:55.704715 29881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.126:45853 every 8 connection(s)
I20260927 04:00:55.705063 29814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
I20260927 04:00:55.706125 29882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:00:55.708441 29882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c3e092adfae4d008e6c22c1c0c22a34: Bootstrap starting.
I20260927 04:00:55.709022 29882 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c3e092adfae4d008e6c22c1c0c22a34: Neither blocks nor log segments found. Creating new log.
I20260927 04:00:55.709300 29882 log.cc:824] T 00000000000000000000000000000000 P 8c3e092adfae4d008e6c22c1c0c22a34: Log is configured to *not* fsync() on all Append() calls
I20260927 04:00:55.709988 29882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c3e092adfae4d008e6c22c1c0c22a34: No bootstrap required, opened a new log
I20260927 04:00:55.711620 29882 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c3e092adfae4d008e6c22c1c0c22a34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c3e092adfae4d008e6c22c1c0c22a34" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45853 } }
I20260927 04:00:55.711764 29882 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c3e092adfae4d008e6c22c1c0c22a34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:00:55.711787 29882 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c3e092adfae4d008e6c22c1c0c22a34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c3e092adfae4d008e6c22c1c0c22a34, State: Initialized, Role: FOLLOWER
I20260927 04:00:55.711963 29882 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c3e092adfae4d008e6c22c1c0c22a34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c3e092adfae4d008e6c22c1c0c22a34" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45853 } }
I20260927 04:00:55.712049 29882 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c3e092adfae4d008e6c22c1c0c22a34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:00:55.712090 29882 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c3e092adfae4d008e6c22c1c0c22a34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:00:55.712141 29882 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c3e092adfae4d008e6c22c1c0c22a34 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:00:55.712940 29882 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c3e092adfae4d008e6c22c1c0c22a34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c3e092adfae4d008e6c22c1c0c22a34" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45853 } }
I20260927 04:00:55.713025 29882 leader_election.cc:302] T 00000000000000000000000000000000 P 8c3e092adfae4d008e6c22c1c0c22a34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c3e092adfae4d008e6c22c1c0c22a34; no voters:
I20260927 04:00:55.713202 29882 leader_election.cc:288] T 00000000000000000000000000000000 P 8c3e092adfae4d008e6c22c1c0c22a34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:00:55.713238 29885 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c3e092adfae4d008e6c22c1c0c22a34 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:00:55.713413 29885 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c3e092adfae4d008e6c22c1c0c22a34 [term 1 LEADER]: Becoming Leader. State: Replica: 8c3e092adfae4d008e6c22c1c0c22a34, State: Running, Role: LEADER
I20260927 04:00:55.713462 29882 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c3e092adfae4d008e6c22c1c0c22a34 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:00:55.714001 29885 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c3e092adfae4d008e6c22c1c0c22a34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c3e092adfae4d008e6c22c1c0c22a34" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45853 } }
I20260927 04:00:55.714643 29886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c3e092adfae4d008e6c22c1c0c22a34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c3e092adfae4d008e6c22c1c0c22a34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c3e092adfae4d008e6c22c1c0c22a34" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45853 } } }
I20260927 04:00:55.714644 29887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c3e092adfae4d008e6c22c1c0c22a34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c3e092adfae4d008e6c22c1c0c22a34. Latest consensus state: current_term: 1 leader_uuid: "8c3e092adfae4d008e6c22c1c0c22a34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c3e092adfae4d008e6c22c1c0c22a34" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45853 } } }
I20260927 04:00:55.714749 29886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c3e092adfae4d008e6c22c1c0c22a34 [sys.catalog]: This master's current role is: LEADER
I20260927 04:00:55.714781 29887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c3e092adfae4d008e6c22c1c0c22a34 [sys.catalog]: This master's current role is: LEADER
I20260927 04:00:55.715189 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 29814
I20260927 04:00:55.715323 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/master-0/wal/instance
I20260927 04:00:55.715546 16101 external_mini_cluster.cc:1428] Setting key a8a44d775242678b0d3f52e84873caff
I20260927 04:00:55.715775 29894 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:00:55.716513 29894 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:00:55.718024 29894 catalog_manager.cc:1347] Generated new cluster ID: 4ef75bee74c94e119a74adff394a9774
I20260927 04:00:55.718077 29894 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 04:00:55 dist-test-slave-glgz krb5kdc[29801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 04:00:55.726512 29904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:55.716286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42334 (local address 127.15.185.126:45853)
0927 04:00:55.717166 (+ 880us) server_negotiation.cc:206] Beginning negotiation
0927 04:00:55.717175 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 04:00:55.717209 (+ 34us) server_negotiation.cc:407] Connection header received
0927 04:00:55.717345 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:00:55.717353 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:00:55.717408 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:00:55.717554 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:00:55.718324 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:55.719388 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:55.720161 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:55.720612 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:55.720735 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:55.723082 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:00:55.723086 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:00:55.723087 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:00:55.723102 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:00:55.724810 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:55.725268 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:55.725272 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:55.725272 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:55.725391 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:55.725608 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:55.725609 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:55.725609 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:55.725870 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:00:55.726035 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 04:00:55.726136 (+ 101us) server_negotiation.cc:299] Negotiation successful
0927 04:00:55.726285 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":834,"thread_start_us":71,"threads_started":1}
I20260927 04:00:55.746587 29894 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:00:55.747402 29894 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:00:55.768445 29894 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c3e092adfae4d008e6c22c1c0c22a34: Generated new TSK 0
I20260927 04:00:55.768707 29894 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.185.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab.
Entry for principal kudu/127.15.185.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab.
I20260927 04:00:55.798631 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.65:0
--local_ip_for_outbound_sockets=127.15.185.65
--webserver_interface=127.15.185.65
--webserver_port=0
--tserver_master_addrs=127.15.185.126:45853
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab
--principal=kudu/127.15.185.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:39093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:00:55.882323 29908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:00:55.882583 29908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:00:55.882627 29908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:00:55.884236 29908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:00:55.884306 29908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:00:55.884369 29908 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.65
I20260927 04:00:55.885942 29908 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:39093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab
--principal=kudu/127.15.185.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.185.65
--webserver_port=0
--tserver_master_addrs=127.15.185.126:45853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29908
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.65
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:00:55.886139 29908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:00:55.886319 29908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:00:55.891376 29913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:00:55.891402 29916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:00:55.891402 29914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:00:55.891849 29908 server_base.cc:1034] running on GCE node
I20260927 04:00:55.892058 29908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:00:55.892336 29908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:00:55.893497 29908 hybrid_clock.cc:648] HybridClock initialized: now 1790481655893485 us; error 30 us; skew 500 ppm
Sep 27 04:00:55 dist-test-slave-glgz krb5kdc[29801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481655, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:00:55.896029 29908 init.cc:377] Logged in from keytab as kudu/127.15.185.65@KRBTEST.COM (short username kudu)
I20260927 04:00:55.896493 29908 webserver.cc:460] Webserver started at http://127.15.185.65:37765/ using document root <none> and password file <none>
I20260927 04:00:55.896736 29908 fs_manager.cc:359] Metadata directory not provided
I20260927 04:00:55.896782 29908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:00:55.896910 29908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:00:55.897959 29908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-0/data/instance:
uuid: "b2154de8c41b4e83bac8daed7caa8973"
format_stamp: "Formatted at 2026-09-27 04:00:55 on dist-test-slave-glgz"
server_key: "97f2390d35e4e67b41727c7fe49bcd58"
server_key_iv: "1ce6a91b6c7c9d179923f02c0a739430"
server_key_version: "encryptionkey@0"
I20260927 04:00:55.898307 29908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-0/wal/instance:
uuid: "b2154de8c41b4e83bac8daed7caa8973"
format_stamp: "Formatted at 2026-09-27 04:00:55 on dist-test-slave-glgz"
server_key: "97f2390d35e4e67b41727c7fe49bcd58"
server_key_iv: "1ce6a91b6c7c9d179923f02c0a739430"
server_key_version: "encryptionkey@0"
I20260927 04:00:55.899760 29908 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 04:00:55.900589 29923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:00:55.900836 29908 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:00:55.900925 29908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-0/wal
uuid: "b2154de8c41b4e83bac8daed7caa8973"
format_stamp: "Formatted at 2026-09-27 04:00:55 on dist-test-slave-glgz"
server_key: "97f2390d35e4e67b41727c7fe49bcd58"
server_key_iv: "1ce6a91b6c7c9d179923f02c0a739430"
server_key_version: "encryptionkey@0"
I20260927 04:00:55.900980 29908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:00:55.910624 29908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:00:55.912737 29908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:00:55.912853 29908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:00:55.913041 29908 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:00:55.913314 29908 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:00:55.913347 29908 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:00:55.913368 29908 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:00:55.913381 29908 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:00:55.919590 29908 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.65:37115
I20260927 04:00:55.919654 30036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.65:37115 every 8 connection(s)
I20260927 04:00:55.919970 29908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
I20260927 04:00:55.924009 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 29908
I20260927 04:00:55.924124 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-0/wal/instance
I20260927 04:00:55.924346 16101 external_mini_cluster.cc:1428] Setting key bdd813271fcecc516b585655ceb1e772
Sep 27 04:00:55 dist-test-slave-glgz krb5kdc[29801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481655, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.65@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 04:00:55.931447 29904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:55.920769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:59757 (local address 127.15.185.126:45853)
0927 04:00:55.920864 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0927 04:00:55.920866 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:00:55.921063 (+ 197us) server_negotiation.cc:407] Connection header received
0927 04:00:55.921639 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:00:55.921640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:00:55.921657 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:00:55.921709 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:00:55.922805 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:55.923451 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:55.924812 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:55.925081 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:55.925168 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:55.927995 (+ 2827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:00:55.927997 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:00:55.927997 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:00:55.928006 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:00:55.929853 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:55.930299 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:55.930301 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:55.930301 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:55.930381 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:55.930630 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:55.930631 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:55.930631 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:55.930800 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:00:55.930881 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 04:00:55.931220 (+ 339us) server_negotiation.cc:299] Negotiation successful
0927 04:00:55.931320 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 04:00:55.931523 30039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:55.920738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45853 (local address 127.15.185.65:59757)
0927 04:00:55.920920 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:55.920970 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 04:00:55.921521 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:00:55.921836 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:00:55.921840 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:00:55.922451 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:00:55.922678 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:55.922683 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:55.923550 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:55.923552 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:55.924724 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:55.924727 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:55.925180 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:55.925182 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:55.925360 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:55.925896 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:00:55.925907 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:00:55.927885 (+ 1978us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:00:55.929962 (+ 2077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:00:55.929970 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:00:55.929972 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:00:55.930216 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:00:55.930457 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:00:55.930458 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:00:55.930458 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:00:55.930565 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:00:55.930877 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:00:55.930883 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:00:55.931015 (+ 132us) client_negotiation.cc:769] Sending connection context
0927 04:00:55.931177 (+ 162us) client_negotiation.cc:240] Negotiation successful
0927 04:00:55.931317 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":83,"threads_started":1}
I20260927 04:00:55.932145 30037 heartbeater.cc:347] Connected to a master server at 127.15.185.126:45853
I20260927 04:00:55.932296 30037 heartbeater.cc:464] Registering TS with master...
I20260927 04:00:55.932533 30037 heartbeater.cc:511] Master 127.15.185.126:45853 requested a full tablet report, sending...
I20260927 04:00:55.933142 29846 ts_manager.cc:194] Registered new tserver with Master: b2154de8c41b4e83bac8daed7caa8973 (127.15.185.65:37115)
WARNING: no policy specified for kudu/127.15.185.66@KRBTEST.COM; defaulting to no policy
I20260927 04:00:55.933864 29846 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.185.65@KRBTEST.COM'} at 127.15.185.65:59757
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab.
Entry for principal kudu/127.15.185.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab.
I20260927 04:00:55.953761 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.66:0
--local_ip_for_outbound_sockets=127.15.185.66
--webserver_interface=127.15.185.66
--webserver_port=0
--tserver_master_addrs=127.15.185.126:45853
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab
--principal=kudu/127.15.185.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:39093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:00:56.035471 30042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:00:56.035698 30042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:00:56.035724 30042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:00:56.036913 30042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:00:56.036962 30042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:00:56.036989 30042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.66
I20260927 04:00:56.038436 30042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:39093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab
--principal=kudu/127.15.185.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.185.66
--webserver_port=0
--tserver_master_addrs=127.15.185.126:45853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.66
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:00:56.038679 30042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:00:56.038913 30042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 04:00:56.044226 30042 server_base.cc:1034] running on GCE node
W20260927 04:00:56.044198 30050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:00:56.044209 30048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:00:56.044209 30047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:00:56.044673 30042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:00:56.044945 30042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:00:56.046124 30042 hybrid_clock.cc:648] HybridClock initialized: now 1790481656046096 us; error 42 us; skew 500 ppm
Sep 27 04:00:56 dist-test-slave-glgz krb5kdc[29801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481656, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:00:56.048681 30042 init.cc:377] Logged in from keytab as kudu/127.15.185.66@KRBTEST.COM (short username kudu)
I20260927 04:00:56.049163 30042 webserver.cc:460] Webserver started at http://127.15.185.66:33105/ using document root <none> and password file <none>
I20260927 04:00:56.049402 30042 fs_manager.cc:359] Metadata directory not provided
I20260927 04:00:56.049449 30042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:00:56.049587 30042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:00:56.050720 30042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-1/data/instance:
uuid: "267c7a7418a74aa8b9e0828d24ed7709"
format_stamp: "Formatted at 2026-09-27 04:00:56 on dist-test-slave-glgz"
server_key: "686c6646d3e856802f2dc7ec5e09dfeb"
server_key_iv: "86d776cdce8de99266ca638d4cc57002"
server_key_version: "encryptionkey@0"
I20260927 04:00:56.051146 30042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-1/wal/instance:
uuid: "267c7a7418a74aa8b9e0828d24ed7709"
format_stamp: "Formatted at 2026-09-27 04:00:56 on dist-test-slave-glgz"
server_key: "686c6646d3e856802f2dc7ec5e09dfeb"
server_key_iv: "86d776cdce8de99266ca638d4cc57002"
server_key_version: "encryptionkey@0"
I20260927 04:00:56.052527 30042 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:00:56.053352 30057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:00:56.053570 30042 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:00:56.053671 30042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-1/wal
uuid: "267c7a7418a74aa8b9e0828d24ed7709"
format_stamp: "Formatted at 2026-09-27 04:00:56 on dist-test-slave-glgz"
server_key: "686c6646d3e856802f2dc7ec5e09dfeb"
server_key_iv: "86d776cdce8de99266ca638d4cc57002"
server_key_version: "encryptionkey@0"
I20260927 04:00:56.053736 30042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:00:56.067692 30042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:00:56.069723 30042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:00:56.069857 30042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:00:56.070063 30042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:00:56.070369 30042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:00:56.070405 30042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:00:56.070429 30042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:00:56.070443 30042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:00:56.076741 30042 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.66:40909
I20260927 04:00:56.076812 30170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.66:40909 every 8 connection(s)
I20260927 04:00:56.077176 30042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
I20260927 04:00:56.078696 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 30042
I20260927 04:00:56.078804 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-1/wal/instance
I20260927 04:00:56.079027 16101 external_mini_cluster.cc:1428] Setting key 42464c6cf9c27caa0507edc67423f5c1
Sep 27 04:00:56 dist-test-slave-glgz krb5kdc[29801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481656, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.66@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 04:00:56.088547 29904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:56.077963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:33733 (local address 127.15.185.126:45853)
0927 04:00:56.078034 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 04:00:56.078037 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:00:56.078233 (+ 196us) server_negotiation.cc:407] Connection header received
0927 04:00:56.078893 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:00:56.078895 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:00:56.078908 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:00:56.078984 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:00:56.080368 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:56.080952 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:56.082172 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:56.082520 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:56.082602 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:56.085072 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:00:56.085073 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:00:56.085074 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:00:56.085085 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:00:56.086917 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:56.087440 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:56.087442 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:56.087442 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:56.087510 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:56.087775 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:56.087777 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:56.087777 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:56.087964 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:00:56.088044 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 04:00:56.088332 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 04:00:56.088440 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 04:00:56.088577 30173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:56.077941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45853 (local address 127.15.185.66:33733)
0927 04:00:56.078100 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:56.078142 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 04:00:56.078760 (+ 618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:00:56.079103 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:00:56.079109 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:00:56.079944 (+ 835us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:00:56.080236 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:56.080244 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:56.081092 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:56.081094 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:56.082028 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:56.082030 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:56.082630 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:56.082631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:56.082775 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:56.083412 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:00:56.083421 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:00:56.084964 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:00:56.087024 (+ 2060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:00:56.087031 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:00:56.087033 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:00:56.087336 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:00:56.087585 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:00:56.087586 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:00:56.087586 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:00:56.087695 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:00:56.088037 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:00:56.088042 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:00:56.088169 (+ 127us) client_negotiation.cc:769] Sending connection context
0927 04:00:56.088253 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 04:00:56.088399 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":84,"threads_started":1}
WARNING: no policy specified for kudu/127.15.185.67@KRBTEST.COM; defaulting to no policy
I20260927 04:00:56.089262 30171 heartbeater.cc:347] Connected to a master server at 127.15.185.126:45853
I20260927 04:00:56.089392 30171 heartbeater.cc:464] Registering TS with master...
I20260927 04:00:56.089591 30171 heartbeater.cc:511] Master 127.15.185.126:45853 requested a full tablet report, sending...
I20260927 04:00:56.090098 29846 ts_manager.cc:194] Registered new tserver with Master: 267c7a7418a74aa8b9e0828d24ed7709 (127.15.185.66:40909)
I20260927 04:00:56.090490 29846 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.185.66@KRBTEST.COM'} at 127.15.185.66:33733
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab.
Entry for principal kudu/127.15.185.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab.
I20260927 04:00:56.106887 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.67:0
--local_ip_for_outbound_sockets=127.15.185.67
--webserver_interface=127.15.185.67
--webserver_port=0
--tserver_master_addrs=127.15.185.126:45853
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab
--principal=kudu/127.15.185.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:39093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:00:56.192310 30176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:00:56.192519 30176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:00:56.192543 30176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:00:56.193670 30176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:00:56.193713 30176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:00:56.193739 30176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.67
I20260927 04:00:56.195201 30176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:39093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab
--principal=kudu/127.15.185.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.185.67
--webserver_port=0
--tserver_master_addrs=127.15.185.126:45853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.67
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:00:56.195389 30176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:00:56.195562 30176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:00:56.201045 30184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:00:56.201049 30182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:00:56.201058 30176 server_base.cc:1034] running on GCE node
W20260927 04:00:56.201045 30181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:00:56.201601 30176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:00:56.201825 30176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:00:56.202978 30176 hybrid_clock.cc:648] HybridClock initialized: now 1790481656202964 us; error 27 us; skew 500 ppm
Sep 27 04:00:56 dist-test-slave-glgz krb5kdc[29801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481656, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:00:56.205523 30176 init.cc:377] Logged in from keytab as kudu/127.15.185.67@KRBTEST.COM (short username kudu)
I20260927 04:00:56.206008 30176 webserver.cc:460] Webserver started at http://127.15.185.67:42267/ using document root <none> and password file <none>
I20260927 04:00:56.206240 30176 fs_manager.cc:359] Metadata directory not provided
I20260927 04:00:56.206311 30176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:00:56.206487 30176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:00:56.207603 30176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-2/data/instance:
uuid: "33e06028c7ef47f8ac35fcafb1372c71"
format_stamp: "Formatted at 2026-09-27 04:00:56 on dist-test-slave-glgz"
server_key: "630f3b08295e9dabe1c90c06649bc480"
server_key_iv: "45015d192198317bba13fdfb3131306e"
server_key_version: "encryptionkey@0"
I20260927 04:00:56.207955 30176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-2/wal/instance:
uuid: "33e06028c7ef47f8ac35fcafb1372c71"
format_stamp: "Formatted at 2026-09-27 04:00:56 on dist-test-slave-glgz"
server_key: "630f3b08295e9dabe1c90c06649bc480"
server_key_iv: "45015d192198317bba13fdfb3131306e"
server_key_version: "encryptionkey@0"
I20260927 04:00:56.209272 30176 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 04:00:56.210116 30191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:00:56.210337 30176 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:00:56.210433 30176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-2/wal
uuid: "33e06028c7ef47f8ac35fcafb1372c71"
format_stamp: "Formatted at 2026-09-27 04:00:56 on dist-test-slave-glgz"
server_key: "630f3b08295e9dabe1c90c06649bc480"
server_key_iv: "45015d192198317bba13fdfb3131306e"
server_key_version: "encryptionkey@0"
I20260927 04:00:56.210505 30176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:00:56.220593 30176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:00:56.222452 30176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:00:56.222609 30176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:00:56.222834 30176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:00:56.223213 30176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:00:56.223273 30176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:00:56.223315 30176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:00:56.223341 30176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:00:56.229583 30176 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.67:39559
I20260927 04:00:56.229637 30304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.67:39559 every 8 connection(s)
I20260927 04:00:56.229981 30176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
I20260927 04:00:56.231937 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 30176
I20260927 04:00:56.232045 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-2/wal/instance
I20260927 04:00:56.232203 16101 external_mini_cluster.cc:1428] Setting key 492511220374b781cbe3262c4eb1eeaa
Sep 27 04:00:56 dist-test-slave-glgz krb5kdc[29801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481656, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.67@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 04:00:56.239786 29904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:56.230747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:47505 (local address 127.15.185.126:45853)
0927 04:00:56.230832 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0927 04:00:56.230834 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:00:56.231024 (+ 190us) server_negotiation.cc:407] Connection header received
0927 04:00:56.231608 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:00:56.231609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:00:56.231621 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:00:56.231673 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:00:56.233076 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:56.233590 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:56.234685 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:56.234859 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:56.234909 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:56.237212 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:00:56.237213 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:00:56.237214 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:00:56.237220 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:00:56.238462 (+ 1242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:56.238882 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:56.238882 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:56.238883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:56.238928 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:56.239193 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:56.239194 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:56.239194 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:56.239307 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:00:56.239362 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 04:00:56.239609 (+ 247us) server_negotiation.cc:299] Negotiation successful
0927 04:00:56.239676 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 04:00:56.240062 30307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:56.230704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45853 (local address 127.15.185.67:47505)
0927 04:00:56.230896 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:56.230935 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 04:00:56.231512 (+ 577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:00:56.231742 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:00:56.231745 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:00:56.232553 (+ 808us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:00:56.232875 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:56.232881 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:56.233687 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:56.233688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:56.234610 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:56.234613 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:56.234929 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:56.234930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:56.235033 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:56.235673 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:00:56.235682 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:00:56.237128 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:00:56.238533 (+ 1405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:00:56.238539 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:00:56.238541 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:00:56.238815 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:00:56.238982 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:00:56.238983 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:00:56.238983 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:00:56.239144 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:00:56.239364 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:00:56.239369 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:00:56.239488 (+ 119us) client_negotiation.cc:769] Sending connection context
0927 04:00:56.239569 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 04:00:56.239772 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":103,"threads_started":1}
I20260927 04:00:56.240710 30305 heartbeater.cc:347] Connected to a master server at 127.15.185.126:45853
I20260927 04:00:56.240841 30305 heartbeater.cc:464] Registering TS with master...
I20260927 04:00:56.241029 30305 heartbeater.cc:511] Master 127.15.185.126:45853 requested a full tablet report, sending...
I20260927 04:00:56.241438 29846 ts_manager.cc:194] Registered new tserver with Master: 33e06028c7ef47f8ac35fcafb1372c71 (127.15.185.67:39559)
I20260927 04:00:56.241860 29846 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.185.67@KRBTEST.COM'} at 127.15.185.67:47505
I20260927 04:00:56.243945 16101 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 04:00:56.251288 29904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:56.245248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42344 (local address 127.15.185.126:45853)
0927 04:00:56.245303 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 04:00:56.245305 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:00:56.245429 (+ 124us) server_negotiation.cc:407] Connection header received
0927 04:00:56.245468 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:00:56.245468 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:00:56.245477 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:00:56.245523 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:00:56.246141 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:56.246682 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:56.247460 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:56.247651 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:56.247722 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:56.248969 (+ 1247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:00:56.248970 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:00:56.248970 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:00:56.248976 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:00:56.250250 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:56.250559 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:56.250560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:56.250560 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:56.250607 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:56.250787 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:56.250787 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:56.250787 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:56.250922 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:00:56.250996 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 04:00:56.251158 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 04:00:56.251227 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 04:00:56.253000 29846 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42344:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 04:00:56.254599 29846 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 04:00:56.261348 30314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:56.257087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.65:37115 (local address 127.0.0.1:49164)
0927 04:00:56.257423 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:56.257433 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 04:00:56.257475 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:00:56.257700 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:00:56.257703 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:00:56.257705 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:00:56.257776 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:56.257778 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:56.259206 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:56.259207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:56.260299 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:56.260302 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:56.261116 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:56.261116 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:56.261171 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:56.261182 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 04:00:56.261237 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 04:00:56.261291 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"mutex_wait_us":2,"thread_start_us":72,"threads_started":1}
I20260927 04:00:56.261415 30318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:56.257363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49164 (local address 127.15.185.65:37115)
0927 04:00:56.257553 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 04:00:56.257557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:00:56.257566 (+ 9us) server_negotiation.cc:407] Connection header received
0927 04:00:56.257600 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:00:56.257602 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:00:56.257614 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:00:56.257681 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:00:56.257908 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:56.259098 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:56.260385 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:56.261036 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:56.261110 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:56.261168 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 04:00:56.261272 (+ 104us) server_negotiation.cc:299] Negotiation successful
0927 04:00:56.261315 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":70,"threads_started":1}
I20260927 04:00:56.261713 30317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:56.257319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53564 (local address 127.15.185.67:39559)
0927 04:00:56.258258 (+ 939us) server_negotiation.cc:206] Beginning negotiation
0927 04:00:56.258262 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:00:56.258268 (+ 6us) server_negotiation.cc:407] Connection header received
0927 04:00:56.258298 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:00:56.258300 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:00:56.258311 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:00:56.258398 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:00:56.258500 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:56.259234 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:56.260628 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:56.261346 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:56.261418 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:56.261474 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 04:00:56.261592 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 04:00:56.261634 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":904,"thread_start_us":93,"threads_started":1}
I20260927 04:00:56.261674 30313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:56.257092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.67:39559 (local address 127.0.0.1:53564)
0927 04:00:56.257446 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:56.257455 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 04:00:56.257496 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:00:56.258398 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:00:56.258398 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:00:56.258400 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:00:56.258442 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:56.258444 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:56.259336 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:56.259337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:56.260541 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:56.260543 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:56.261438 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:56.261439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:56.261514 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:56.261520 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 04:00:56.261578 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 04:00:56.261614 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":35,"thread_start_us":66,"threads_started":1}
I20260927 04:00:56.262542 30315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:56.257085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.66:40909 (local address 127.0.0.1:40260)
0927 04:00:56.257205 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:56.257246 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 04:00:56.257321 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:00:56.258080 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:00:56.258081 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:00:56.258085 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:00:56.258134 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:56.258135 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:56.259390 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:56.259391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:56.260541 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:56.260543 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:56.262309 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:56.262311 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:56.262373 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:56.262378 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 04:00:56.262436 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 04:00:56.262475 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"mutex_wait_us":11,"thread_start_us":57,"threads_started":1}
I20260927 04:00:56.262564 30316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:56.257269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40260 (local address 127.15.185.66:40909)
0927 04:00:56.257832 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0927 04:00:56.257836 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:00:56.257845 (+ 9us) server_negotiation.cc:407] Connection header received
0927 04:00:56.257898 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:00:56.257900 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:00:56.257914 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:00:56.257983 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:00:56.258220 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:56.259278 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:56.261513 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:56.262233 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:56.262310 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:56.262369 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 04:00:56.262448 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 04:00:56.262488 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":89,"threads_started":1}
I20260927 04:00:56.262571 29971 tablet_service.cc:1467] Processing CreateTablet for tablet 217c94be83a7475da8c06e78c19dc716 (DEFAULT_TABLE table=test-table [id=a61bbb1d93af4d8091fb94a6589fd92c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:00:56.262756 30239 tablet_service.cc:1467] Processing CreateTablet for tablet 217c94be83a7475da8c06e78c19dc716 (DEFAULT_TABLE table=test-table [id=a61bbb1d93af4d8091fb94a6589fd92c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:00:56.262951 29971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 217c94be83a7475da8c06e78c19dc716. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:00:56.263078 30239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 217c94be83a7475da8c06e78c19dc716. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:00:56.263531 30105 tablet_service.cc:1467] Processing CreateTablet for tablet 217c94be83a7475da8c06e78c19dc716 (DEFAULT_TABLE table=test-table [id=a61bbb1d93af4d8091fb94a6589fd92c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:00:56.263852 30105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 217c94be83a7475da8c06e78c19dc716. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:00:56.266480 30319 tablet_bootstrap.cc:492] T 217c94be83a7475da8c06e78c19dc716 P 33e06028c7ef47f8ac35fcafb1372c71: Bootstrap starting.
I20260927 04:00:56.267122 30319 tablet_bootstrap.cc:654] T 217c94be83a7475da8c06e78c19dc716 P 33e06028c7ef47f8ac35fcafb1372c71: Neither blocks nor log segments found. Creating new log.
I20260927 04:00:56.267241 30320 tablet_bootstrap.cc:492] T 217c94be83a7475da8c06e78c19dc716 P 267c7a7418a74aa8b9e0828d24ed7709: Bootstrap starting.
I20260927 04:00:56.267634 30319 log.cc:824] T 217c94be83a7475da8c06e78c19dc716 P 33e06028c7ef47f8ac35fcafb1372c71: Log is configured to *not* fsync() on all Append() calls
I20260927 04:00:56.267865 30320 tablet_bootstrap.cc:654] T 217c94be83a7475da8c06e78c19dc716 P 267c7a7418a74aa8b9e0828d24ed7709: Neither blocks nor log segments found. Creating new log.
I20260927 04:00:56.268323 30320 log.cc:824] T 217c94be83a7475da8c06e78c19dc716 P 267c7a7418a74aa8b9e0828d24ed7709: Log is configured to *not* fsync() on all Append() calls
I20260927 04:00:56.268373 30319 tablet_bootstrap.cc:492] T 217c94be83a7475da8c06e78c19dc716 P 33e06028c7ef47f8ac35fcafb1372c71: No bootstrap required, opened a new log
I20260927 04:00:56.268442 30319 ts_tablet_manager.cc:1397] T 217c94be83a7475da8c06e78c19dc716 P 33e06028c7ef47f8ac35fcafb1372c71: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 04:00:56.268648 30321 tablet_bootstrap.cc:492] T 217c94be83a7475da8c06e78c19dc716 P b2154de8c41b4e83bac8daed7caa8973: Bootstrap starting.
I20260927 04:00:56.269043 30320 tablet_bootstrap.cc:492] T 217c94be83a7475da8c06e78c19dc716 P 267c7a7418a74aa8b9e0828d24ed7709: No bootstrap required, opened a new log
I20260927 04:00:56.269110 30320 ts_tablet_manager.cc:1397] T 217c94be83a7475da8c06e78c19dc716 P 267c7a7418a74aa8b9e0828d24ed7709: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 04:00:56.269379 30321 tablet_bootstrap.cc:654] T 217c94be83a7475da8c06e78c19dc716 P b2154de8c41b4e83bac8daed7caa8973: Neither blocks nor log segments found. Creating new log.
I20260927 04:00:56.269901 30321 log.cc:824] T 217c94be83a7475da8c06e78c19dc716 P b2154de8c41b4e83bac8daed7caa8973: Log is configured to *not* fsync() on all Append() calls
I20260927 04:00:56.270542 30321 tablet_bootstrap.cc:492] T 217c94be83a7475da8c06e78c19dc716 P b2154de8c41b4e83bac8daed7caa8973: No bootstrap required, opened a new log
I20260927 04:00:56.270458 30319 raft_consensus.cc:348] T 217c94be83a7475da8c06e78c19dc716 P 33e06028c7ef47f8ac35fcafb1372c71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2154de8c41b4e83bac8daed7caa8973" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 37115 } } peers { permanent_uuid: "267c7a7418a74aa8b9e0828d24ed7709" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 40909 } } peers { permanent_uuid: "33e06028c7ef47f8ac35fcafb1372c71" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 39559 } }
I20260927 04:00:56.270618 30321 ts_tablet_manager.cc:1397] T 217c94be83a7475da8c06e78c19dc716 P b2154de8c41b4e83bac8daed7caa8973: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 04:00:56.270618 30319 raft_consensus.cc:374] T 217c94be83a7475da8c06e78c19dc716 P 33e06028c7ef47f8ac35fcafb1372c71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:00:56.270653 30319 raft_consensus.cc:729] T 217c94be83a7475da8c06e78c19dc716 P 33e06028c7ef47f8ac35fcafb1372c71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33e06028c7ef47f8ac35fcafb1372c71, State: Initialized, Role: FOLLOWER
I20260927 04:00:56.270849 30319 consensus_queue.cc:260] T 217c94be83a7475da8c06e78c19dc716 P 33e06028c7ef47f8ac35fcafb1372c71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2154de8c41b4e83bac8daed7caa8973" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 37115 } } peers { permanent_uuid: "267c7a7418a74aa8b9e0828d24ed7709" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 40909 } } peers { permanent_uuid: "33e06028c7ef47f8ac35fcafb1372c71" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 39559 } }
I20260927 04:00:56.271190 30319 ts_tablet_manager.cc:1428] T 217c94be83a7475da8c06e78c19dc716 P 33e06028c7ef47f8ac35fcafb1372c71: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 04:00:56.271229 30305 heartbeater.cc:503] Master 127.15.185.126:45853 was elected leader, sending a full tablet report...
I20260927 04:00:56.271106 30320 raft_consensus.cc:348] T 217c94be83a7475da8c06e78c19dc716 P 267c7a7418a74aa8b9e0828d24ed7709 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2154de8c41b4e83bac8daed7caa8973" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 37115 } } peers { permanent_uuid: "267c7a7418a74aa8b9e0828d24ed7709" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 40909 } } peers { permanent_uuid: "33e06028c7ef47f8ac35fcafb1372c71" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 39559 } }
I20260927 04:00:56.271292 30320 raft_consensus.cc:374] T 217c94be83a7475da8c06e78c19dc716 P 267c7a7418a74aa8b9e0828d24ed7709 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:00:56.271341 30320 raft_consensus.cc:729] T 217c94be83a7475da8c06e78c19dc716 P 267c7a7418a74aa8b9e0828d24ed7709 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 267c7a7418a74aa8b9e0828d24ed7709, State: Initialized, Role: FOLLOWER
I20260927 04:00:56.271447 30320 consensus_queue.cc:260] T 217c94be83a7475da8c06e78c19dc716 P 267c7a7418a74aa8b9e0828d24ed7709 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2154de8c41b4e83bac8daed7caa8973" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 37115 } } peers { permanent_uuid: "267c7a7418a74aa8b9e0828d24ed7709" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 40909 } } peers { permanent_uuid: "33e06028c7ef47f8ac35fcafb1372c71" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 39559 } }
I20260927 04:00:56.271704 30320 ts_tablet_manager.cc:1428] T 217c94be83a7475da8c06e78c19dc716 P 267c7a7418a74aa8b9e0828d24ed7709: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 04:00:56.271838 30171 heartbeater.cc:503] Master 127.15.185.126:45853 was elected leader, sending a full tablet report...
I20260927 04:00:56.273379 30321 raft_consensus.cc:348] T 217c94be83a7475da8c06e78c19dc716 P b2154de8c41b4e83bac8daed7caa8973 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2154de8c41b4e83bac8daed7caa8973" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 37115 } } peers { permanent_uuid: "267c7a7418a74aa8b9e0828d24ed7709" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 40909 } } peers { permanent_uuid: "33e06028c7ef47f8ac35fcafb1372c71" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 39559 } }
I20260927 04:00:56.273523 30321 raft_consensus.cc:374] T 217c94be83a7475da8c06e78c19dc716 P b2154de8c41b4e83bac8daed7caa8973 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:00:56.273561 30321 raft_consensus.cc:729] T 217c94be83a7475da8c06e78c19dc716 P b2154de8c41b4e83bac8daed7caa8973 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2154de8c41b4e83bac8daed7caa8973, State: Initialized, Role: FOLLOWER
I20260927 04:00:56.273715 30321 consensus_queue.cc:260] T 217c94be83a7475da8c06e78c19dc716 P b2154de8c41b4e83bac8daed7caa8973 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2154de8c41b4e83bac8daed7caa8973" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 37115 } } peers { permanent_uuid: "267c7a7418a74aa8b9e0828d24ed7709" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 40909 } } peers { permanent_uuid: "33e06028c7ef47f8ac35fcafb1372c71" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 39559 } }
I20260927 04:00:56.273993 30321 ts_tablet_manager.cc:1428] T 217c94be83a7475da8c06e78c19dc716 P b2154de8c41b4e83bac8daed7caa8973: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 04:00:56.274051 30037 heartbeater.cc:503] Master 127.15.185.126:45853 was elected leader, sending a full tablet report...
I20260927 04:00:56.306133 30327 raft_consensus.cc:482] T 217c94be83a7475da8c06e78c19dc716 P b2154de8c41b4e83bac8daed7caa8973 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:00:56.306257 30327 raft_consensus.cc:504] T 217c94be83a7475da8c06e78c19dc716 P b2154de8c41b4e83bac8daed7caa8973 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2154de8c41b4e83bac8daed7caa8973" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 37115 } } peers { permanent_uuid: "267c7a7418a74aa8b9e0828d24ed7709" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 40909 } } peers { permanent_uuid: "33e06028c7ef47f8ac35fcafb1372c71" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 39559 } }
I20260927 04:00:56.306766 30327 leader_election.cc:288] T 217c94be83a7475da8c06e78c19dc716 P b2154de8c41b4e83bac8daed7caa8973 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 267c7a7418a74aa8b9e0828d24ed7709 (127.15.185.66:40909), 33e06028c7ef47f8ac35fcafb1372c71 (127.15.185.67:39559)
I20260927 04:00:56.309590 30039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:56.306959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.67:39559 (local address 127.15.185.65:55425)
0927 04:00:56.307025 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:56.307038 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 04:00:56.307128 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:00:56.307295 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:00:56.307296 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:00:56.307297 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:00:56.307372 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:56.307374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:56.308033 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:56.308034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:56.308772 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:56.308774 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:56.309320 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:56.309321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:56.309401 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:56.309407 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 04:00:56.309478 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 04:00:56.309519 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 04:00:56.309590 30317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:56.307033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:55425 (local address 127.15.185.67:39559)
0927 04:00:56.307092 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 04:00:56.307094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:00:56.307108 (+ 14us) server_negotiation.cc:407] Connection header received
0927 04:00:56.307208 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:00:56.307209 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:00:56.307218 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:00:56.307292 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:00:56.307453 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:56.307937 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:56.308866 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:56.309239 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:56.309323 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:56.309332 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 04:00:56.309484 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 04:00:56.309521 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 04:00:56.309880 30316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:56.307010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:48683 (local address 127.15.185.66:40909)
0927 04:00:56.307083 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 04:00:56.307085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:00:56.307347 (+ 262us) server_negotiation.cc:407] Connection header received
0927 04:00:56.307457 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:00:56.307458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:00:56.307468 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:00:56.307518 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:00:56.307748 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:56.308208 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:56.309109 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:56.309492 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:56.309578 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:56.309585 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0927 04:00:56.309748 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 04:00:56.309802 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 04:00:56.309947 30259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217c94be83a7475da8c06e78c19dc716" candidate_uuid: "b2154de8c41b4e83bac8daed7caa8973" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33e06028c7ef47f8ac35fcafb1372c71" is_pre_election: true
I20260927 04:00:56.310141 30259 raft_consensus.cc:2454] T 217c94be83a7475da8c06e78c19dc716 P 33e06028c7ef47f8ac35fcafb1372c71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2154de8c41b4e83bac8daed7caa8973 in term 0.
I20260927 04:00:56.310137 30328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:56.306960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.66:40909 (local address 127.15.185.65:48683)
0927 04:00:56.307267 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:56.307281 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 04:00:56.307358 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:00:56.307521 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:00:56.307521 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:00:56.307523 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:00:56.307602 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:56.307604 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:56.308292 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:56.308293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:56.309018 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:56.309019 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:56.309578 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:56.309579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:56.309654 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:56.309658 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 04:00:56.309991 (+ 333us) client_negotiation.cc:240] Negotiation successful
0927 04:00:56.310044 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":98,"thread_start_us":150,"threads_started":1}
I20260927 04:00:56.310403 29925 leader_election.cc:302] T 217c94be83a7475da8c06e78c19dc716 P b2154de8c41b4e83bac8daed7caa8973 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33e06028c7ef47f8ac35fcafb1372c71, b2154de8c41b4e83bac8daed7caa8973; no voters:
I20260927 04:00:56.310549 30125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217c94be83a7475da8c06e78c19dc716" candidate_uuid: "b2154de8c41b4e83bac8daed7caa8973" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "267c7a7418a74aa8b9e0828d24ed7709" is_pre_election: true
I20260927 04:00:56.310601 30327 raft_consensus.cc:2781] T 217c94be83a7475da8c06e78c19dc716 P b2154de8c41b4e83bac8daed7caa8973 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 04:00:56.310662 30327 raft_consensus.cc:482] T 217c94be83a7475da8c06e78c19dc716 P b2154de8c41b4e83bac8daed7caa8973 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 04:00:56.310685 30327 raft_consensus.cc:3037] T 217c94be83a7475da8c06e78c19dc716 P b2154de8c41b4e83bac8daed7caa8973 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:00:56.310724 30125 raft_consensus.cc:2454] T 217c94be83a7475da8c06e78c19dc716 P 267c7a7418a74aa8b9e0828d24ed7709 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2154de8c41b4e83bac8daed7caa8973 in term 0.
I20260927 04:00:56.311524 30327 raft_consensus.cc:504] T 217c94be83a7475da8c06e78c19dc716 P b2154de8c41b4e83bac8daed7caa8973 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2154de8c41b4e83bac8daed7caa8973" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 37115 } } peers { permanent_uuid: "267c7a7418a74aa8b9e0828d24ed7709" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 40909 } } peers { permanent_uuid: "33e06028c7ef47f8ac35fcafb1372c71" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 39559 } }
I20260927 04:00:56.311689 30327 leader_election.cc:288] T 217c94be83a7475da8c06e78c19dc716 P b2154de8c41b4e83bac8daed7caa8973 [CANDIDATE]: Term 1 election: Requested vote from peers 267c7a7418a74aa8b9e0828d24ed7709 (127.15.185.66:40909), 33e06028c7ef47f8ac35fcafb1372c71 (127.15.185.67:39559)
I20260927 04:00:56.311913 30259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217c94be83a7475da8c06e78c19dc716" candidate_uuid: "b2154de8c41b4e83bac8daed7caa8973" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33e06028c7ef47f8ac35fcafb1372c71"
I20260927 04:00:56.311924 30125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217c94be83a7475da8c06e78c19dc716" candidate_uuid: "b2154de8c41b4e83bac8daed7caa8973" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "267c7a7418a74aa8b9e0828d24ed7709"
I20260927 04:00:56.312001 30125 raft_consensus.cc:3037] T 217c94be83a7475da8c06e78c19dc716 P 267c7a7418a74aa8b9e0828d24ed7709 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:00:56.312016 30259 raft_consensus.cc:3037] T 217c94be83a7475da8c06e78c19dc716 P 33e06028c7ef47f8ac35fcafb1372c71 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:00:56.312747 30125 raft_consensus.cc:2454] T 217c94be83a7475da8c06e78c19dc716 P 267c7a7418a74aa8b9e0828d24ed7709 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2154de8c41b4e83bac8daed7caa8973 in term 1.
I20260927 04:00:56.312748 30259 raft_consensus.cc:2454] T 217c94be83a7475da8c06e78c19dc716 P 33e06028c7ef47f8ac35fcafb1372c71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2154de8c41b4e83bac8daed7caa8973 in term 1.
I20260927 04:00:56.312916 29925 leader_election.cc:302] T 217c94be83a7475da8c06e78c19dc716 P b2154de8c41b4e83bac8daed7caa8973 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33e06028c7ef47f8ac35fcafb1372c71, b2154de8c41b4e83bac8daed7caa8973; no voters:
I20260927 04:00:56.313040 30327 raft_consensus.cc:2781] T 217c94be83a7475da8c06e78c19dc716 P b2154de8c41b4e83bac8daed7caa8973 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:00:56.313205 30327 raft_consensus.cc:686] T 217c94be83a7475da8c06e78c19dc716 P b2154de8c41b4e83bac8daed7caa8973 [term 1 LEADER]: Becoming Leader. State: Replica: b2154de8c41b4e83bac8daed7caa8973, State: Running, Role: LEADER
I20260927 04:00:56.313282 30327 consensus_queue.cc:237] T 217c94be83a7475da8c06e78c19dc716 P b2154de8c41b4e83bac8daed7caa8973 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2154de8c41b4e83bac8daed7caa8973" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 37115 } } peers { permanent_uuid: "267c7a7418a74aa8b9e0828d24ed7709" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 40909 } } peers { permanent_uuid: "33e06028c7ef47f8ac35fcafb1372c71" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 39559 } }
I20260927 04:00:56.314265 29845 catalog_manager.cc:5495] T 217c94be83a7475da8c06e78c19dc716 P b2154de8c41b4e83bac8daed7caa8973 reported cstate change: term changed from 0 to 1, leader changed from <none> to b2154de8c41b4e83bac8daed7caa8973 (127.15.185.65). New cstate: current_term: 1 leader_uuid: "b2154de8c41b4e83bac8daed7caa8973" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2154de8c41b4e83bac8daed7caa8973" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 37115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "267c7a7418a74aa8b9e0828d24ed7709" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 40909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33e06028c7ef47f8ac35fcafb1372c71" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 39559 } health_report { overall_health: UNKNOWN } } }
I20260927 04:00:56.319949 30318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:56.317362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49176 (local address 127.15.185.65:37115)
0927 04:00:56.317424 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 04:00:56.317426 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:00:56.317435 (+ 9us) server_negotiation.cc:407] Connection header received
0927 04:00:56.317465 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:00:56.317466 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:00:56.317483 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:00:56.317569 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:00:56.317713 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:56.318192 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:56.318961 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:56.319263 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:56.319347 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:56.319522 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:00:56.319632 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:00:56.319733 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 04:00:56.319816 (+ 83us) server_negotiation.cc:299] Negotiation successful
0927 04:00:56.319840 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 04:00:56.321868 30259 raft_consensus.cc:1261] T 217c94be83a7475da8c06e78c19dc716 P 33e06028c7ef47f8ac35fcafb1372c71 [term 1 FOLLOWER]: Refusing update from remote peer b2154de8c41b4e83bac8daed7caa8973: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:00:56.321871 30125 raft_consensus.cc:1261] T 217c94be83a7475da8c06e78c19dc716 P 267c7a7418a74aa8b9e0828d24ed7709 [term 1 FOLLOWER]: Refusing update from remote peer b2154de8c41b4e83bac8daed7caa8973: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:00:56.322237 30327 consensus_queue.cc:1035] T 217c94be83a7475da8c06e78c19dc716 P b2154de8c41b4e83bac8daed7caa8973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "267c7a7418a74aa8b9e0828d24ed7709" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 40909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:00:56.322376 30329 consensus_queue.cc:1035] T 217c94be83a7475da8c06e78c19dc716 P b2154de8c41b4e83bac8daed7caa8973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33e06028c7ef47f8ac35fcafb1372c71" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 39559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:00:56.324756 30332 mvcc.cc:205] Tried to move back new op lower bound from 7333812864291520512 to 7333812864259973120. Current Snapshot: MvccSnapshot[applied={T|T < 7333812864291520512}]
I20260927 04:00:56.327226 29846 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:00:56.327330 29846 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:00:56.328132 29846 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8c3e092adfae4d008e6c22c1c0c22a34: Sending DeleteTablet for 3 replicas of tablet 217c94be83a7475da8c06e78c19dc716
I20260927 04:00:56.328507 30105 tablet_service.cc:1514] Processing DeleteTablet for tablet 217c94be83a7475da8c06e78c19dc716 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:00:56 UTC) from {username='kudu', principal='kudu/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:40260
I20260927 04:00:56.328557 30239 tablet_service.cc:1514] Processing DeleteTablet for tablet 217c94be83a7475da8c06e78c19dc716 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:00:56 UTC) from {username='kudu', principal='kudu/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:53564
I20260927 04:00:56.328572 29971 tablet_service.cc:1514] Processing DeleteTablet for tablet 217c94be83a7475da8c06e78c19dc716 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:00:56 UTC) from {username='kudu', principal='kudu/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:49164
I20260927 04:00:56.328758 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 29908
I20260927 04:00:56.328786 30344 tablet_replica.cc:330] T 217c94be83a7475da8c06e78c19dc716 P b2154de8c41b4e83bac8daed7caa8973: stopping tablet replica
I20260927 04:00:56.328888 30344 raft_consensus.cc:2229] T 217c94be83a7475da8c06e78c19dc716 P b2154de8c41b4e83bac8daed7caa8973 [term 1 LEADER]: Raft consensus shutting down.
I20260927 04:00:56.328981 30344 raft_consensus.cc:2258] T 217c94be83a7475da8c06e78c19dc716 P b2154de8c41b4e83bac8daed7caa8973 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:00:56.334704 30342 tablet_replica.cc:330] T 217c94be83a7475da8c06e78c19dc716 P 267c7a7418a74aa8b9e0828d24ed7709: stopping tablet replica
W20260927 04:00:56.334697 29830 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.185.65:37115 (error 108)
I20260927 04:00:56.334761 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 30042
I20260927 04:00:56.334836 30342 raft_consensus.cc:2229] T 217c94be83a7475da8c06e78c19dc716 P 267c7a7418a74aa8b9e0828d24ed7709 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:00:56.335004 30342 raft_consensus.cc:2258] T 217c94be83a7475da8c06e78c19dc716 P 267c7a7418a74aa8b9e0828d24ed7709 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:00:56.335218 30343 tablet_replica.cc:330] T 217c94be83a7475da8c06e78c19dc716 P 33e06028c7ef47f8ac35fcafb1372c71: stopping tablet replica
I20260927 04:00:56.335351 30343 raft_consensus.cc:2229] T 217c94be83a7475da8c06e78c19dc716 P 33e06028c7ef47f8ac35fcafb1372c71 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:00:56.335474 30343 raft_consensus.cc:2258] T 217c94be83a7475da8c06e78c19dc716 P 33e06028c7ef47f8ac35fcafb1372c71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:00:56.336210 30343 ts_tablet_manager.cc:1905] T 217c94be83a7475da8c06e78c19dc716 P 33e06028c7ef47f8ac35fcafb1372c71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:00:56.336724 30315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:56.336584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.65:37115 (local address 127.0.0.1:49180)
0927 04:00:56.336631 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:56.336687 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.65:37115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260927 04:00:56.337018 29830 catalog_manager.cc:4584] TS b2154de8c41b4e83bac8daed7caa8973 (127.15.185.65:37115): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 217c94be83a7475da8c06e78c19dc716: Network error: Client connection negotiation failed: client connection to 127.15.185.65:37115: connect: Connection refused (error 111)
I20260927 04:00:56.338115 30343 ts_tablet_manager.cc:1918] T 217c94be83a7475da8c06e78c19dc716 P 33e06028c7ef47f8ac35fcafb1372c71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:00:56.338210 30343 log.cc:1196] T 217c94be83a7475da8c06e78c19dc716 P 33e06028c7ef47f8ac35fcafb1372c71: Deleting WAL directory at /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-2/wal/wals/217c94be83a7475da8c06e78c19dc716
I20260927 04:00:56.338479 30343 ts_tablet_manager.cc:1939] T 217c94be83a7475da8c06e78c19dc716 P 33e06028c7ef47f8ac35fcafb1372c71: Deleting consensus metadata
I20260927 04:00:56.338898 29831 catalog_manager.cc:4841] TS 33e06028c7ef47f8ac35fcafb1372c71 (127.15.185.67:39559): tablet 217c94be83a7475da8c06e78c19dc716 (table test-table [id=a61bbb1d93af4d8091fb94a6589fd92c]) successfully deleted
I20260927 04:00:56.340026 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 30176
I20260927 04:00:56.340082 30315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:56.339959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.66:40909 (local address 127.0.0.1:40268)
0927 04:00:56.340028 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:56.340056 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.66:40909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 04:00:56.345216 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 29814
I20260927 04:00:56.351080 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:45853
--webserver_interface=127.15.185.126
--webserver_port=33989
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:39093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.185.126:45853
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:00:56.433465 30346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:00:56.433708 30346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:00:56.433733 30346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:00:56.434885 30346 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:00:56.434921 30346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:00:56.434935 30346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:00:56.434947 30346 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:00:56.434960 30346 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:00:56.436539 30346 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:39093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.15.185.126:45853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:45853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.185.126
--webserver_port=33989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:00:56.436736 30346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:00:56.436925 30346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:00:56.445531 30352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:00:56.445644 30346 server_base.cc:1034] running on GCE node
W20260927 04:00:56.445580 30353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:00:56.445530 30355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:00:56.445995 30346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:00:56.446323 30346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:00:56.447484 30346 hybrid_clock.cc:648] HybridClock initialized: now 1790481656447468 us; error 40 us; skew 500 ppm
Sep 27 04:00:56 dist-test-slave-glgz krb5kdc[29801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481656, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:00:56.456597 30346 init.cc:377] Logged in from keytab as kudu/127.15.185.126@KRBTEST.COM (short username kudu)
I20260927 04:00:56.457116 30346 webserver.cc:460] Webserver started at http://127.15.185.126:33989/ using document root <none> and password file <none>
I20260927 04:00:56.457377 30346 fs_manager.cc:359] Metadata directory not provided
I20260927 04:00:56.457460 30346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:00:56.458961 30346 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 04:00:56.459874 30362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:00:56.460090 30346 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:00:56.460210 30346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/master-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/master-0/wal
uuid: "8c3e092adfae4d008e6c22c1c0c22a34"
format_stamp: "Formatted at 2026-09-27 04:00:55 on dist-test-slave-glgz"
server_key: "828e675d78684da1271578c26259e0d5"
server_key_iv: "6c8a661f1af8a136fb39b7dc3eae775d"
server_key_version: "encryptionkey@0"
I20260927 04:00:56.460758 30346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:00:56.470867 30346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:00:56.472626 30346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:00:56.472735 30346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:00:56.476722 30346 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.126:45853
I20260927 04:00:56.476758 30414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.126:45853 every 8 connection(s)
I20260927 04:00:56.477169 30346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
I20260927 04:00:56.478820 30415 sys_catalog.cc:263] Verifying existing consensus state
I20260927 04:00:56.479576 30415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c3e092adfae4d008e6c22c1c0c22a34: Bootstrap starting.
I20260927 04:00:56.487007 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 30346
I20260927 04:00:56.487380 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.65:37115
--local_ip_for_outbound_sockets=127.15.185.65
--tserver_master_addrs=127.15.185.126:45853
--webserver_port=37765
--webserver_interface=127.15.185.65
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab
--principal=kudu/127.15.185.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:39093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 04:00:56.487864 30415 log.cc:824] T 00000000000000000000000000000000 P 8c3e092adfae4d008e6c22c1c0c22a34: Log is configured to *not* fsync() on all Append() calls
I20260927 04:00:56.489868 30415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c3e092adfae4d008e6c22c1c0c22a34: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 04:00:56.490160 30415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c3e092adfae4d008e6c22c1c0c22a34: Bootstrap complete.
I20260927 04:00:56.492551 30415 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c3e092adfae4d008e6c22c1c0c22a34 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c3e092adfae4d008e6c22c1c0c22a34" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45853 } }
I20260927 04:00:56.492801 30415 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c3e092adfae4d008e6c22c1c0c22a34 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c3e092adfae4d008e6c22c1c0c22a34, State: Initialized, Role: FOLLOWER
I20260927 04:00:56.493013 30415 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c3e092adfae4d008e6c22c1c0c22a34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c3e092adfae4d008e6c22c1c0c22a34" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45853 } }
I20260927 04:00:56.493111 30415 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c3e092adfae4d008e6c22c1c0c22a34 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:00:56.493170 30415 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c3e092adfae4d008e6c22c1c0c22a34 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:00:56.493218 30415 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c3e092adfae4d008e6c22c1c0c22a34 [term 1 FOLLOWER]: Advancing to term 2
I20260927 04:00:56.494280 30415 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c3e092adfae4d008e6c22c1c0c22a34 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c3e092adfae4d008e6c22c1c0c22a34" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45853 } }
I20260927 04:00:56.494410 30415 leader_election.cc:302] T 00000000000000000000000000000000 P 8c3e092adfae4d008e6c22c1c0c22a34 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c3e092adfae4d008e6c22c1c0c22a34; no voters:
I20260927 04:00:56.494611 30415 leader_election.cc:288] T 00000000000000000000000000000000 P 8c3e092adfae4d008e6c22c1c0c22a34 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 04:00:56.494647 30419 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c3e092adfae4d008e6c22c1c0c22a34 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 04:00:56.494751 30419 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c3e092adfae4d008e6c22c1c0c22a34 [term 2 LEADER]: Becoming Leader. State: Replica: 8c3e092adfae4d008e6c22c1c0c22a34, State: Running, Role: LEADER
I20260927 04:00:56.494886 30415 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c3e092adfae4d008e6c22c1c0c22a34 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:00:56.494915 30419 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c3e092adfae4d008e6c22c1c0c22a34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c3e092adfae4d008e6c22c1c0c22a34" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45853 } }
I20260927 04:00:56.495308 30420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c3e092adfae4d008e6c22c1c0c22a34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8c3e092adfae4d008e6c22c1c0c22a34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c3e092adfae4d008e6c22c1c0c22a34" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45853 } } }
I20260927 04:00:56.495313 30421 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c3e092adfae4d008e6c22c1c0c22a34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c3e092adfae4d008e6c22c1c0c22a34. Latest consensus state: current_term: 2 leader_uuid: "8c3e092adfae4d008e6c22c1c0c22a34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c3e092adfae4d008e6c22c1c0c22a34" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45853 } } }
I20260927 04:00:56.495415 30420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c3e092adfae4d008e6c22c1c0c22a34 [sys.catalog]: This master's current role is: LEADER
I20260927 04:00:56.495430 30421 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c3e092adfae4d008e6c22c1c0c22a34 [sys.catalog]: This master's current role is: LEADER
I20260927 04:00:56.496119 30426 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:00:56.497109 30426 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:00:56.497228 30426 catalog_manager.cc:1259] Loaded cluster ID: 4ef75bee74c94e119a74adff394a9774
I20260927 04:00:56.497268 30426 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:00:56.497532 30426 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260927 04:00:56.502838 30426 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260927 04:00:56.505535 30426 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:00:56.505740 30426 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8c3e092adfae4d008e6c22c1c0c22a34: Loaded TSK: 0
I20260927 04:00:56.505954 30426 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 04:00:56.572306 30418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:00:56.572510 30418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:00:56.572533 30418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:00:56.573679 30418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:00:56.573720 30418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:00:56.573747 30418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.65
I20260927 04:00:56.575230 30418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:39093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab
--principal=kudu/127.15.185.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.65:37115
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.185.65
--webserver_port=37765
--tserver_master_addrs=127.15.185.126:45853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.65
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:00:56.575471 30418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:00:56.575706 30418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:00:56.581970 30443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:00:56.581998 30445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:00:56.582062 30442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:00:56.582404 30418 server_base.cc:1034] running on GCE node
I20260927 04:00:56.582619 30418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:00:56.582896 30418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:00:56.584074 30418 hybrid_clock.cc:648] HybridClock initialized: now 1790481656584039 us; error 46 us; skew 500 ppm
Sep 27 04:00:56 dist-test-slave-glgz krb5kdc[29801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481656, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:00:56.586853 30418 init.cc:377] Logged in from keytab as kudu/127.15.185.65@KRBTEST.COM (short username kudu)
I20260927 04:00:56.587384 30418 webserver.cc:460] Webserver started at http://127.15.185.65:37765/ using document root <none> and password file <none>
I20260927 04:00:56.587628 30418 fs_manager.cc:359] Metadata directory not provided
I20260927 04:00:56.587802 30418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:00:56.589389 30418 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 04:00:56.590219 30452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:00:56.590444 30418 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:00:56.590552 30418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-0/wal
uuid: "b2154de8c41b4e83bac8daed7caa8973"
format_stamp: "Formatted at 2026-09-27 04:00:55 on dist-test-slave-glgz"
server_key: "97f2390d35e4e67b41727c7fe49bcd58"
server_key_iv: "1ce6a91b6c7c9d179923f02c0a739430"
server_key_version: "encryptionkey@0"
I20260927 04:00:56.590967 30418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:00:56.628736 30418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:00:56.630733 30418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:00:56.630903 30418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:00:56.631181 30418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:00:56.631702 30459 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 04:00:56.633229 30418 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 04:00:56.633402 30418 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 04:00:56.633473 30418 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 04:00:56.634544 30418 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 04:00:56.634586 30418 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 04:00:56.634719 30459 tablet_bootstrap.cc:492] T 217c94be83a7475da8c06e78c19dc716 P b2154de8c41b4e83bac8daed7caa8973: Bootstrap starting.
I20260927 04:00:56.641111 30418 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.65:37115
I20260927 04:00:56.641212 30566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.65:37115 every 8 connection(s)
I20260927 04:00:56.641535 30418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
I20260927 04:00:56.642998 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 30418
I20260927 04:00:56.643316 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.66:40909
--local_ip_for_outbound_sockets=127.15.185.66
--tserver_master_addrs=127.15.185.126:45853
--webserver_port=33105
--webserver_interface=127.15.185.66
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab
--principal=kudu/127.15.185.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:39093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 04:00:56.646298 30459 log.cc:824] T 217c94be83a7475da8c06e78c19dc716 P b2154de8c41b4e83bac8daed7caa8973: Log is configured to *not* fsync() on all Append() calls
I20260927 04:00:56.649569 30459 tablet_bootstrap.cc:492] T 217c94be83a7475da8c06e78c19dc716 P b2154de8c41b4e83bac8daed7caa8973: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 04:00:56.649981 30459 tablet_bootstrap.cc:492] T 217c94be83a7475da8c06e78c19dc716 P b2154de8c41b4e83bac8daed7caa8973: Bootstrap complete.
Sep 27 04:00:56 dist-test-slave-glgz krb5kdc[29801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481656, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.65@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 04:00:56.651410 30459 ts_tablet_manager.cc:1397] T 217c94be83a7475da8c06e78c19dc716 P b2154de8c41b4e83bac8daed7caa8973: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.011s
I20260927 04:00:56.653298 30459 raft_consensus.cc:348] T 217c94be83a7475da8c06e78c19dc716 P b2154de8c41b4e83bac8daed7caa8973 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2154de8c41b4e83bac8daed7caa8973" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 37115 } } peers { permanent_uuid: "267c7a7418a74aa8b9e0828d24ed7709" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 40909 } } peers { permanent_uuid: "33e06028c7ef47f8ac35fcafb1372c71" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 39559 } }
I20260927 04:00:56.653827 30459 raft_consensus.cc:729] T 217c94be83a7475da8c06e78c19dc716 P b2154de8c41b4e83bac8daed7caa8973 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2154de8c41b4e83bac8daed7caa8973, State: Initialized, Role: FOLLOWER
I20260927 04:00:56.654027 30459 consensus_queue.cc:260] T 217c94be83a7475da8c06e78c19dc716 P b2154de8c41b4e83bac8daed7caa8973 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2154de8c41b4e83bac8daed7caa8973" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 37115 } } peers { permanent_uuid: "267c7a7418a74aa8b9e0828d24ed7709" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 40909 } } peers { permanent_uuid: "33e06028c7ef47f8ac35fcafb1372c71" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 39559 } }
I20260927 04:00:56.654389 30459 ts_tablet_manager.cc:1428] T 217c94be83a7475da8c06e78c19dc716 P b2154de8c41b4e83bac8daed7caa8973: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260927 04:00:56.655246 30569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:56.642407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45853 (local address 127.15.185.65:50537)
0927 04:00:56.642650 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:56.642706 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 04:00:56.643555 (+ 849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:00:56.644871 (+ 1316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:00:56.644876 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:00:56.645397 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:00:56.645759 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:56.645767 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:56.647109 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:56.647110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:56.648013 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:56.648017 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:56.648678 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:56.648679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:56.648827 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:56.649388 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:00:56.649397 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:00:56.651417 (+ 2020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:00:56.653438 (+ 2021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:00:56.653444 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:00:56.653447 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:00:56.653681 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:00:56.653997 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:00:56.653998 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:00:56.653999 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:00:56.654100 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:00:56.654629 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:00:56.654634 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:00:56.654739 (+ 105us) client_negotiation.cc:769] Sending connection context
0927 04:00:56.654839 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 04:00:56.655022 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":109,"threads_started":1}
I20260927 04:00:56.655247 30570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:56.642553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:50537 (local address 127.15.185.126:45853)
0927 04:00:56.642705 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 04:00:56.642714 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 04:00:56.642824 (+ 110us) server_negotiation.cc:407] Connection header received
0927 04:00:56.643821 (+ 997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:00:56.643829 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:00:56.643894 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:00:56.644824 (+ 930us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:00:56.645880 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:56.646974 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:56.648132 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:56.648547 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:56.648682 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:56.651578 (+ 2896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:00:56.651582 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:00:56.651582 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:00:56.651595 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:00:56.653343 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:56.653785 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:56.653788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:56.653789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:56.653919 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:56.654195 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:56.654196 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:56.654196 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:56.654528 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:00:56.654600 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 04:00:56.654907 (+ 307us) server_negotiation.cc:299] Negotiation successful
0927 04:00:56.655037 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":77,"threads_started":1}
I20260927 04:00:56.655910 30567 heartbeater.cc:347] Connected to a master server at 127.15.185.126:45853
I20260927 04:00:56.656003 30567 heartbeater.cc:464] Registering TS with master...
I20260927 04:00:56.656194 30567 heartbeater.cc:511] Master 127.15.185.126:45853 requested a full tablet report, sending...
I20260927 04:00:56.656886 30379 ts_manager.cc:194] Registered new tserver with Master: b2154de8c41b4e83bac8daed7caa8973 (127.15.185.65:37115)
I20260927 04:00:56.657966 30379 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.185.65@KRBTEST.COM'} at 127.15.185.65:50537
I20260927 04:00:56.658916 30567 heartbeater.cc:503] Master 127.15.185.126:45853 was elected leader, sending a full tablet report...
Sep 27 04:00:56 dist-test-slave-glgz krb5kdc[29801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481656, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.126@KRBTEST.COM for kudu/127.15.185.65@KRBTEST.COM
I20260927 04:00:56.666708 30575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:56.657552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49184 (local address 127.15.185.65:37115)
0927 04:00:56.657727 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 04:00:56.657732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:00:56.657938 (+ 206us) server_negotiation.cc:407] Connection header received
0927 04:00:56.658031 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:00:56.658034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:00:56.658051 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:00:56.658123 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:00:56.658691 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:56.660019 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:56.660741 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:56.661239 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:56.661315 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:56.663618 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:00:56.663625 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:00:56.663625 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:00:56.663631 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:00:56.665186 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:56.665843 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:56.665846 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:56.665847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:56.665911 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:56.666125 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:56.666126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:56.666126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:56.666297 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:00:56.666359 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 04:00:56.666563 (+ 204us) server_negotiation.cc:299] Negotiation successful
0927 04:00:56.666640 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":85,"threads_started":1}
I20260927 04:00:56.666874 30574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:56.657470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.65:37115 (local address 127.0.0.1:49184)
0927 04:00:56.657805 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:56.657852 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 04:00:56.657939 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:00:56.658189 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:00:56.658193 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:00:56.658538 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:00:56.658595 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:56.658598 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:56.660101 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:56.660102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:56.660662 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:56.660664 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:56.661323 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:56.661324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:56.661379 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:56.661901 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:00:56.661906 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:00:56.663527 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:00:56.665273 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:00:56.665280 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:00:56.665283 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:00:56.665602 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:00:56.665981 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:00:56.665982 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:00:56.665982 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:00:56.666059 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:00:56.666361 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:00:56.666369 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:00:56.666428 (+ 59us) client_negotiation.cc:769] Sending connection context
0927 04:00:56.666540 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 04:00:56.666682 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":92,"threads_started":1}
I20260927 04:00:56.667357 30501 tablet_service.cc:1514] Processing DeleteTablet for tablet 217c94be83a7475da8c06e78c19dc716 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:00:56 UTC) from {username='kudu', principal='kudu/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:49184
I20260927 04:00:56.667894 30576 tablet_replica.cc:330] stopping tablet replica
I20260927 04:00:56.668018 30576 raft_consensus.cc:2229] T 217c94be83a7475da8c06e78c19dc716 P b2154de8c41b4e83bac8daed7caa8973 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:00:56.668100 30576 raft_consensus.cc:2258] T 217c94be83a7475da8c06e78c19dc716 P b2154de8c41b4e83bac8daed7caa8973 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:00:56.668471 30576 ts_tablet_manager.cc:1905] T 217c94be83a7475da8c06e78c19dc716 P b2154de8c41b4e83bac8daed7caa8973: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:00:56.670439 30576 ts_tablet_manager.cc:1918] T 217c94be83a7475da8c06e78c19dc716 P b2154de8c41b4e83bac8daed7caa8973: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:00:56.670524 30576 log.cc:1196] T 217c94be83a7475da8c06e78c19dc716 P b2154de8c41b4e83bac8daed7caa8973: Deleting WAL directory at /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-0/wal/wals/217c94be83a7475da8c06e78c19dc716
I20260927 04:00:56.670766 30576 ts_tablet_manager.cc:1939] T 217c94be83a7475da8c06e78c19dc716 P b2154de8c41b4e83bac8daed7caa8973: Deleting consensus metadata
I20260927 04:00:56.671347 30363 catalog_manager.cc:4841] TS b2154de8c41b4e83bac8daed7caa8973 (127.15.185.65:37115): tablet 217c94be83a7475da8c06e78c19dc716 (table test-table [id=a61bbb1d93af4d8091fb94a6589fd92c]) successfully deleted
W20260927 04:00:56.741456 30571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:00:56.741686 30571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:00:56.741712 30571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:00:56.743010 30571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:00:56.743085 30571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:00:56.743117 30571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.66
I20260927 04:00:56.744798 30571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:39093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab
--principal=kudu/127.15.185.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.66:40909
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.185.66
--webserver_port=33105
--tserver_master_addrs=127.15.185.126:45853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.66
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:00:56.745087 30571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:00:56.745347 30571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:00:56.750874 30582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:00:56.750903 30584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:00:56.750869 30581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:00:56.750923 30571 server_base.cc:1034] running on GCE node
I20260927 04:00:56.751356 30571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:00:56.751595 30571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:00:56.752759 30571 hybrid_clock.cc:648] HybridClock initialized: now 1790481656752756 us; error 37 us; skew 500 ppm
Sep 27 04:00:56 dist-test-slave-glgz krb5kdc[29801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481656, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:00:56.755354 30571 init.cc:377] Logged in from keytab as kudu/127.15.185.66@KRBTEST.COM (short username kudu)
I20260927 04:00:56.755878 30571 webserver.cc:460] Webserver started at http://127.15.185.66:33105/ using document root <none> and password file <none>
I20260927 04:00:56.756125 30571 fs_manager.cc:359] Metadata directory not provided
I20260927 04:00:56.756217 30571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:00:56.757639 30571 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 04:00:56.758349 30591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:00:56.758526 30571 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 04:00:56.758638 30571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-1/wal
uuid: "267c7a7418a74aa8b9e0828d24ed7709"
format_stamp: "Formatted at 2026-09-27 04:00:56 on dist-test-slave-glgz"
server_key: "686c6646d3e856802f2dc7ec5e09dfeb"
server_key_iv: "86d776cdce8de99266ca638d4cc57002"
server_key_version: "encryptionkey@0"
I20260927 04:00:56.759018 30571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:00:56.777436 30571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:00:56.779536 30571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:00:56.779721 30571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:00:56.779975 30571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:00:56.780436 30598 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 04:00:56.781842 30571 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 04:00:56.781922 30571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 04:00:56.781975 30571 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 04:00:56.782719 30571 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 04:00:56.782799 30571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 04:00:56.782804 30598 tablet_bootstrap.cc:492] T 217c94be83a7475da8c06e78c19dc716 P 267c7a7418a74aa8b9e0828d24ed7709: Bootstrap starting.
I20260927 04:00:56.789742 30571 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.66:40909
I20260927 04:00:56.789829 30705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.66:40909 every 8 connection(s)
I20260927 04:00:56.790165 30571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
I20260927 04:00:56.790568 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 30571
I20260927 04:00:56.790935 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.67:39559
--local_ip_for_outbound_sockets=127.15.185.67
--tserver_master_addrs=127.15.185.126:45853
--webserver_port=42267
--webserver_interface=127.15.185.67
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab
--principal=kudu/127.15.185.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:39093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 04:00:56.793804 30598 log.cc:824] T 217c94be83a7475da8c06e78c19dc716 P 267c7a7418a74aa8b9e0828d24ed7709: Log is configured to *not* fsync() on all Append() calls
I20260927 04:00:56.796463 30598 tablet_bootstrap.cc:492] T 217c94be83a7475da8c06e78c19dc716 P 267c7a7418a74aa8b9e0828d24ed7709: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 04:00:56.796802 30598 tablet_bootstrap.cc:492] T 217c94be83a7475da8c06e78c19dc716 P 267c7a7418a74aa8b9e0828d24ed7709: Bootstrap complete.
I20260927 04:00:56.798033 30598 ts_tablet_manager.cc:1397] T 217c94be83a7475da8c06e78c19dc716 P 267c7a7418a74aa8b9e0828d24ed7709: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.009s
Sep 27 04:00:56 dist-test-slave-glgz krb5kdc[29801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481656, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.66@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 04:00:56.799469 30598 raft_consensus.cc:348] T 217c94be83a7475da8c06e78c19dc716 P 267c7a7418a74aa8b9e0828d24ed7709 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2154de8c41b4e83bac8daed7caa8973" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 37115 } } peers { permanent_uuid: "267c7a7418a74aa8b9e0828d24ed7709" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 40909 } } peers { permanent_uuid: "33e06028c7ef47f8ac35fcafb1372c71" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 39559 } }
I20260927 04:00:56.799757 30598 raft_consensus.cc:729] T 217c94be83a7475da8c06e78c19dc716 P 267c7a7418a74aa8b9e0828d24ed7709 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 267c7a7418a74aa8b9e0828d24ed7709, State: Initialized, Role: FOLLOWER
I20260927 04:00:56.799911 30598 consensus_queue.cc:260] T 217c94be83a7475da8c06e78c19dc716 P 267c7a7418a74aa8b9e0828d24ed7709 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2154de8c41b4e83bac8daed7caa8973" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 37115 } } peers { permanent_uuid: "267c7a7418a74aa8b9e0828d24ed7709" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 40909 } } peers { permanent_uuid: "33e06028c7ef47f8ac35fcafb1372c71" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 39559 } }
I20260927 04:00:56.800205 30598 ts_tablet_manager.cc:1428] T 217c94be83a7475da8c06e78c19dc716 P 267c7a7418a74aa8b9e0828d24ed7709: Time spent starting tablet: real 0.002s user 0.003s sys 0.002s
I20260927 04:00:56.802858 30708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:56.790954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45853 (local address 127.15.185.66:41227)
0927 04:00:56.791392 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:56.791456 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 04:00:56.792222 (+ 766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:00:56.792627 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:00:56.792632 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:00:56.793128 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:00:56.793467 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:56.793474 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:56.794658 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:56.794659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:56.796027 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:56.796029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:56.796703 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:56.796704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:56.796869 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:56.797502 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:00:56.797513 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:00:56.799337 (+ 1824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:00:56.801466 (+ 2129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:00:56.801472 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:00:56.801474 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:00:56.801693 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:00:56.801947 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:00:56.801947 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:00:56.801947 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:00:56.802050 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:00:56.802388 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:00:56.802392 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:00:56.802507 (+ 115us) client_negotiation.cc:769] Sending connection context
0927 04:00:56.802583 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 04:00:56.802712 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":118,"threads_started":1}
I20260927 04:00:56.802927 30570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:56.791271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:41227 (local address 127.15.185.126:45853)
0927 04:00:56.791357 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0927 04:00:56.791360 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:00:56.791561 (+ 201us) server_negotiation.cc:407] Connection header received
0927 04:00:56.792388 (+ 827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:00:56.792391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:00:56.792406 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:00:56.792441 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:00:56.793575 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:56.794535 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:56.796129 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:56.796572 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:56.796684 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:56.799459 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:00:56.799461 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:00:56.799462 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:00:56.799471 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:00:56.801379 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:56.801790 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:56.801791 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:56.801792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:56.801874 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:56.802134 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:56.802135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:56.802135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:56.802313 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:00:56.802398 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 04:00:56.802669 (+ 271us) server_negotiation.cc:299] Negotiation successful
0927 04:00:56.802776 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 04:00:56.803514 30706 heartbeater.cc:347] Connected to a master server at 127.15.185.126:45853
I20260927 04:00:56.803611 30706 heartbeater.cc:464] Registering TS with master...
I20260927 04:00:56.803800 30706 heartbeater.cc:511] Master 127.15.185.126:45853 requested a full tablet report, sending...
I20260927 04:00:56.804328 30379 ts_manager.cc:194] Registered new tserver with Master: 267c7a7418a74aa8b9e0828d24ed7709 (127.15.185.66:40909)
I20260927 04:00:56.804777 30379 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.185.66@KRBTEST.COM'} at 127.15.185.66:41227
I20260927 04:00:56.805433 30706 heartbeater.cc:503] Master 127.15.185.126:45853 was elected leader, sending a full tablet report...
I20260927 04:00:56.808988 30712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:56.804891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40274 (local address 127.15.185.66:40909)
0927 04:00:56.805586 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0927 04:00:56.805591 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:00:56.805601 (+ 10us) server_negotiation.cc:407] Connection header received
0927 04:00:56.805635 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:00:56.805637 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:00:56.805653 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:00:56.805736 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:00:56.805907 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:56.806542 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:56.808159 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:56.808610 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:56.808668 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:56.808704 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0927 04:00:56.808909 (+ 205us) server_negotiation.cc:299] Negotiation successful
0927 04:00:56.808940 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":102,"threads_started":1}
I20260927 04:00:56.809017 30574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:56.804648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.66:40909 (local address 127.0.0.1:40274)
0927 04:00:56.804763 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:56.804775 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 04:00:56.804815 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:00:56.805769 (+ 954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:00:56.805771 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:00:56.805773 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:00:56.805839 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:56.805841 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:56.806617 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:56.806618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:56.808060 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:56.808064 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:56.808695 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:56.808696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:56.808799 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:56.808827 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 04:00:56.808891 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 04:00:56.808938 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89}
I20260927 04:00:56.809412 30640 tablet_service.cc:1514] Processing DeleteTablet for tablet 217c94be83a7475da8c06e78c19dc716 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:00:56 UTC) from {username='kudu', principal='kudu/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:40274
I20260927 04:00:56.810036 30713 tablet_replica.cc:330] stopping tablet replica
I20260927 04:00:56.810138 30713 raft_consensus.cc:2229] T 217c94be83a7475da8c06e78c19dc716 P 267c7a7418a74aa8b9e0828d24ed7709 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:00:56.810209 30713 raft_consensus.cc:2258] T 217c94be83a7475da8c06e78c19dc716 P 267c7a7418a74aa8b9e0828d24ed7709 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:00:56.810523 30713 ts_tablet_manager.cc:1905] T 217c94be83a7475da8c06e78c19dc716 P 267c7a7418a74aa8b9e0828d24ed7709: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:00:56.812733 30713 ts_tablet_manager.cc:1918] T 217c94be83a7475da8c06e78c19dc716 P 267c7a7418a74aa8b9e0828d24ed7709: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:00:56.812796 30713 log.cc:1196] T 217c94be83a7475da8c06e78c19dc716 P 267c7a7418a74aa8b9e0828d24ed7709: Deleting WAL directory at /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-1/wal/wals/217c94be83a7475da8c06e78c19dc716
I20260927 04:00:56.813031 30713 ts_tablet_manager.cc:1939] T 217c94be83a7475da8c06e78c19dc716 P 267c7a7418a74aa8b9e0828d24ed7709: Deleting consensus metadata
I20260927 04:00:56.813534 30366 catalog_manager.cc:4841] TS 267c7a7418a74aa8b9e0828d24ed7709 (127.15.185.66:40909): tablet 217c94be83a7475da8c06e78c19dc716 (table test-table [id=a61bbb1d93af4d8091fb94a6589fd92c]) successfully deleted
W20260927 04:00:56.880968 30709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:00:56.881187 30709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:00:56.881213 30709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:00:56.882279 30709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:00:56.882313 30709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:00:56.882339 30709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.67
I20260927 04:00:56.883765 30709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:39093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab
--principal=kudu/127.15.185.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.67:39559
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.185.67
--webserver_port=42267
--tserver_master_addrs=127.15.185.126:45853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.67
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:00:56.883965 30709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:00:56.884136 30709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:00:56.889175 30719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:00:56.889274 30709 server_base.cc:1034] running on GCE node
W20260927 04:00:56.889153 30718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:00:56.889149 30721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:00:56.889575 30709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:00:56.889815 30709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:00:56.890957 30709 hybrid_clock.cc:648] HybridClock initialized: now 1790481656890928 us; error 41 us; skew 500 ppm
Sep 27 04:00:56 dist-test-slave-glgz krb5kdc[29801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481656, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:00:56.893368 30709 init.cc:377] Logged in from keytab as kudu/127.15.185.67@KRBTEST.COM (short username kudu)
I20260927 04:00:56.893950 30709 webserver.cc:460] Webserver started at http://127.15.185.67:42267/ using document root <none> and password file <none>
I20260927 04:00:56.894164 30709 fs_manager.cc:359] Metadata directory not provided
I20260927 04:00:56.894219 30709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:00:56.895709 30709 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 04:00:56.896299 30728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:00:56.896515 30709 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:00:56.896605 30709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-2/wal
uuid: "33e06028c7ef47f8ac35fcafb1372c71"
format_stamp: "Formatted at 2026-09-27 04:00:56 on dist-test-slave-glgz"
server_key: "630f3b08295e9dabe1c90c06649bc480"
server_key_iv: "45015d192198317bba13fdfb3131306e"
server_key_version: "encryptionkey@0"
I20260927 04:00:56.896965 30709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:00:56.914037 30709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:00:56.916277 30709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:00:56.916465 30709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:00:56.916783 30709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:00:56.917145 30709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:00:56.917208 30709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:00:56.917268 30709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:00:56.917292 30709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:00:56.923506 30709 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.67:39559
I20260927 04:00:56.923595 30841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.67:39559 every 8 connection(s)
I20260927 04:00:56.923987 30709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
I20260927 04:00:56.926592 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 30709
Sep 27 04:00:56 dist-test-slave-glgz krb5kdc[29801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481656, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.67@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 04:00:56.934176 30846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:56.927000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42350 (local address 127.15.185.126:45853)
0927 04:00:56.927245 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0927 04:00:56.927246 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 04:00:56.927251 (+ 5us) server_negotiation.cc:407] Connection header received
0927 04:00:56.927268 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:00:56.927268 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:00:56.927276 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:00:56.927323 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:00:56.928060 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:56.928522 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:56.929555 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:56.929843 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:56.929898 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:56.931086 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:00:56.931088 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:00:56.931088 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:00:56.931095 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:00:56.932858 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:56.933218 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:56.933219 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:56.933219 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:56.933279 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:56.933572 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:56.933573 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:56.933574 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:56.933715 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:00:56.933894 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 04:00:56.933993 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 04:00:56.934077 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":79,"threads_started":1}
I20260927 04:00:56.934710 30844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:56.924774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45853 (local address 127.15.185.67:59181)
0927 04:00:56.924950 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:56.924993 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 04:00:56.925471 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:00:56.925791 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:00:56.925795 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:00:56.926378 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:00:56.926705 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:56.926713 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:56.927675 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:56.927676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:56.928709 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:56.928712 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:56.929074 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:56.929075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:56.929233 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:56.929803 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:00:56.929812 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:00:56.931428 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:00:56.933037 (+ 1609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:00:56.933042 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:00:56.933044 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:00:56.933331 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:00:56.933571 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:00:56.933572 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:00:56.933572 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:00:56.933722 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:00:56.934016 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:00:56.934021 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:00:56.934177 (+ 156us) client_negotiation.cc:769] Sending connection context
0927 04:00:56.934295 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 04:00:56.934470 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":68,"threads_started":1}
I20260927 04:00:56.935068 30570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:56.924838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:59181 (local address 127.15.185.126:45853)
0927 04:00:56.924896 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 04:00:56.924898 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:00:56.925107 (+ 209us) server_negotiation.cc:407] Connection header received
0927 04:00:56.925566 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:00:56.925567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:00:56.925579 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:00:56.925713 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:00:56.926820 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:56.927547 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:56.928800 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:56.928986 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:56.929036 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:56.931525 (+ 2489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:00:56.931527 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:00:56.931527 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:00:56.931534 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:00:56.932952 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:56.933421 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:56.933422 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:56.933422 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:56.933484 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:56.933798 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:56.933799 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:56.933800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:56.933941 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:00:56.934009 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 04:00:56.934875 (+ 866us) server_negotiation.cc:299] Negotiation successful
0927 04:00:56.934959 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 04:00:56.935359 30842 heartbeater.cc:347] Connected to a master server at 127.15.185.126:45853
I20260927 04:00:56.935451 30842 heartbeater.cc:464] Registering TS with master...
I20260927 04:00:56.935662 30842 heartbeater.cc:511] Master 127.15.185.126:45853 requested a full tablet report, sending...
I20260927 04:00:56.936048 30379 ts_manager.cc:194] Registered new tserver with Master: 33e06028c7ef47f8ac35fcafb1372c71 (127.15.185.67:39559)
I20260927 04:00:56.936412 30379 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.185.67@KRBTEST.COM'} at 127.15.185.67:59181
I20260927 04:00:56.946045 16101 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 04:00:56.949028 30570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:56.946400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42364 (local address 127.15.185.126:45853)
0927 04:00:56.946459 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 04:00:56.946461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:00:56.946583 (+ 122us) server_negotiation.cc:407] Connection header received
0927 04:00:56.946648 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:00:56.946648 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:00:56.946683 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:00:56.946780 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:00:56.946914 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:56.947512 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:56.948258 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:56.948442 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:56.948513 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:56.948665 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:00:56.948775 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:00:56.948840 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 04:00:56.948884 (+ 44us) server_negotiation.cc:299] Negotiation successful
0927 04:00:56.948901 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 04:00:56.949325 30379 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:42364:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 04:00:56.951001 30379 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 04:00:56.953611 30640 tablet_service.cc:1467] Processing CreateTablet for tablet b00a70de47d6453fa4864cf5d69157bd (DEFAULT_TABLE table=test-table [id=29cffce467df4fb79421ad074981b8d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:00:56.953619 30501 tablet_service.cc:1467] Processing CreateTablet for tablet b00a70de47d6453fa4864cf5d69157bd (DEFAULT_TABLE table=test-table [id=29cffce467df4fb79421ad074981b8d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:00:56.953836 30640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b00a70de47d6453fa4864cf5d69157bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:00:56.953838 30501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b00a70de47d6453fa4864cf5d69157bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:00:56.955224 30598 tablet_bootstrap.cc:492] T b00a70de47d6453fa4864cf5d69157bd P 267c7a7418a74aa8b9e0828d24ed7709: Bootstrap starting.
I20260927 04:00:56.955502 30459 tablet_bootstrap.cc:492] T b00a70de47d6453fa4864cf5d69157bd P b2154de8c41b4e83bac8daed7caa8973: Bootstrap starting.
I20260927 04:00:56.955567 30598 tablet_bootstrap.cc:654] T b00a70de47d6453fa4864cf5d69157bd P 267c7a7418a74aa8b9e0828d24ed7709: Neither blocks nor log segments found. Creating new log.
I20260927 04:00:56.955799 30459 tablet_bootstrap.cc:654] T b00a70de47d6453fa4864cf5d69157bd P b2154de8c41b4e83bac8daed7caa8973: Neither blocks nor log segments found. Creating new log.
I20260927 04:00:56.956154 30598 tablet_bootstrap.cc:492] T b00a70de47d6453fa4864cf5d69157bd P 267c7a7418a74aa8b9e0828d24ed7709: No bootstrap required, opened a new log
I20260927 04:00:56.956211 30598 ts_tablet_manager.cc:1397] T b00a70de47d6453fa4864cf5d69157bd P 267c7a7418a74aa8b9e0828d24ed7709: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260927 04:00:56.956326 30598 raft_consensus.cc:348] T b00a70de47d6453fa4864cf5d69157bd P 267c7a7418a74aa8b9e0828d24ed7709 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33e06028c7ef47f8ac35fcafb1372c71" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 39559 } } peers { permanent_uuid: "b2154de8c41b4e83bac8daed7caa8973" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 37115 } } peers { permanent_uuid: "267c7a7418a74aa8b9e0828d24ed7709" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 40909 } }
I20260927 04:00:56.956399 30598 raft_consensus.cc:374] T b00a70de47d6453fa4864cf5d69157bd P 267c7a7418a74aa8b9e0828d24ed7709 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:00:56.956450 30598 raft_consensus.cc:729] T b00a70de47d6453fa4864cf5d69157bd P 267c7a7418a74aa8b9e0828d24ed7709 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 267c7a7418a74aa8b9e0828d24ed7709, State: Initialized, Role: FOLLOWER
I20260927 04:00:56.956501 30598 consensus_queue.cc:260] T b00a70de47d6453fa4864cf5d69157bd P 267c7a7418a74aa8b9e0828d24ed7709 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33e06028c7ef47f8ac35fcafb1372c71" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 39559 } } peers { permanent_uuid: "b2154de8c41b4e83bac8daed7caa8973" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 37115 } } peers { permanent_uuid: "267c7a7418a74aa8b9e0828d24ed7709" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 40909 } }
I20260927 04:00:56.956601 30598 ts_tablet_manager.cc:1428] T b00a70de47d6453fa4864cf5d69157bd P 267c7a7418a74aa8b9e0828d24ed7709: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 04:00:56.956730 30459 tablet_bootstrap.cc:492] T b00a70de47d6453fa4864cf5d69157bd P b2154de8c41b4e83bac8daed7caa8973: No bootstrap required, opened a new log
I20260927 04:00:56.956772 30459 ts_tablet_manager.cc:1397] T b00a70de47d6453fa4864cf5d69157bd P b2154de8c41b4e83bac8daed7caa8973: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 04:00:56.956900 30459 raft_consensus.cc:348] T b00a70de47d6453fa4864cf5d69157bd P b2154de8c41b4e83bac8daed7caa8973 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33e06028c7ef47f8ac35fcafb1372c71" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 39559 } } peers { permanent_uuid: "b2154de8c41b4e83bac8daed7caa8973" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 37115 } } peers { permanent_uuid: "267c7a7418a74aa8b9e0828d24ed7709" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 40909 } }
I20260927 04:00:56.956969 30459 raft_consensus.cc:374] T b00a70de47d6453fa4864cf5d69157bd P b2154de8c41b4e83bac8daed7caa8973 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:00:56.957013 30459 raft_consensus.cc:729] T b00a70de47d6453fa4864cf5d69157bd P b2154de8c41b4e83bac8daed7caa8973 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2154de8c41b4e83bac8daed7caa8973, State: Initialized, Role: FOLLOWER
I20260927 04:00:56.957072 30459 consensus_queue.cc:260] T b00a70de47d6453fa4864cf5d69157bd P b2154de8c41b4e83bac8daed7caa8973 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33e06028c7ef47f8ac35fcafb1372c71" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 39559 } } peers { permanent_uuid: "b2154de8c41b4e83bac8daed7caa8973" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 37115 } } peers { permanent_uuid: "267c7a7418a74aa8b9e0828d24ed7709" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 40909 } }
I20260927 04:00:56.957168 30459 ts_tablet_manager.cc:1428] T b00a70de47d6453fa4864cf5d69157bd P b2154de8c41b4e83bac8daed7caa8973: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 04:00:56.957674 30574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:56.953331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.67:39559 (local address 127.0.0.1:53572)
0927 04:00:56.953380 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:56.953391 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 04:00:56.953426 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:00:56.953970 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:00:56.953971 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:00:56.953973 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:00:56.954020 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:56.954022 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:56.955331 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:56.955332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:56.956369 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:56.956372 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:56.957434 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:56.957435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:56.957500 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:56.957504 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 04:00:56.957556 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 04:00:56.957600 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 04:00:56.957923 30848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:56.953479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53572 (local address 127.15.185.67:39559)
0927 04:00:56.953765 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0927 04:00:56.953770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:00:56.953781 (+ 11us) server_negotiation.cc:407] Connection header received
0927 04:00:56.953826 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:00:56.953828 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:00:56.953844 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:00:56.953965 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:00:56.954119 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:56.955206 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:56.956492 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:56.957216 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:56.957390 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:56.957462 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 04:00:56.957782 (+ 320us) server_negotiation.cc:299] Negotiation successful
0927 04:00:56.957827 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":111,"threads_started":1}
I20260927 04:00:56.958540 30776 tablet_service.cc:1467] Processing CreateTablet for tablet b00a70de47d6453fa4864cf5d69157bd (DEFAULT_TABLE table=test-table [id=29cffce467df4fb79421ad074981b8d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:00:56.958827 30776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b00a70de47d6453fa4864cf5d69157bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:00:56.961676 30851 tablet_bootstrap.cc:492] T b00a70de47d6453fa4864cf5d69157bd P 33e06028c7ef47f8ac35fcafb1372c71: Bootstrap starting.
I20260927 04:00:56.962262 30851 tablet_bootstrap.cc:654] T b00a70de47d6453fa4864cf5d69157bd P 33e06028c7ef47f8ac35fcafb1372c71: Neither blocks nor log segments found. Creating new log.
I20260927 04:00:56.962894 30851 log.cc:824] T b00a70de47d6453fa4864cf5d69157bd P 33e06028c7ef47f8ac35fcafb1372c71: Log is configured to *not* fsync() on all Append() calls
I20260927 04:00:56.963696 30851 tablet_bootstrap.cc:492] T b00a70de47d6453fa4864cf5d69157bd P 33e06028c7ef47f8ac35fcafb1372c71: No bootstrap required, opened a new log
I20260927 04:00:56.963752 30851 ts_tablet_manager.cc:1397] T b00a70de47d6453fa4864cf5d69157bd P 33e06028c7ef47f8ac35fcafb1372c71: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 04:00:56.965238 30851 raft_consensus.cc:348] T b00a70de47d6453fa4864cf5d69157bd P 33e06028c7ef47f8ac35fcafb1372c71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33e06028c7ef47f8ac35fcafb1372c71" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 39559 } } peers { permanent_uuid: "b2154de8c41b4e83bac8daed7caa8973" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 37115 } } peers { permanent_uuid: "267c7a7418a74aa8b9e0828d24ed7709" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 40909 } }
I20260927 04:00:56.965376 30851 raft_consensus.cc:374] T b00a70de47d6453fa4864cf5d69157bd P 33e06028c7ef47f8ac35fcafb1372c71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:00:56.965399 30851 raft_consensus.cc:729] T b00a70de47d6453fa4864cf5d69157bd P 33e06028c7ef47f8ac35fcafb1372c71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33e06028c7ef47f8ac35fcafb1372c71, State: Initialized, Role: FOLLOWER
I20260927 04:00:56.965483 30851 consensus_queue.cc:260] T b00a70de47d6453fa4864cf5d69157bd P 33e06028c7ef47f8ac35fcafb1372c71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33e06028c7ef47f8ac35fcafb1372c71" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 39559 } } peers { permanent_uuid: "b2154de8c41b4e83bac8daed7caa8973" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 37115 } } peers { permanent_uuid: "267c7a7418a74aa8b9e0828d24ed7709" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 40909 } }
I20260927 04:00:56.965693 30851 ts_tablet_manager.cc:1428] T b00a70de47d6453fa4864cf5d69157bd P 33e06028c7ef47f8ac35fcafb1372c71: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 04:00:56.965787 30842 heartbeater.cc:503] Master 127.15.185.126:45853 was elected leader, sending a full tablet report...
W20260927 04:00:57.043985 30707 tablet.cc:2369] T b00a70de47d6453fa4864cf5d69157bd P 267c7a7418a74aa8b9e0828d24ed7709: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 04:00:57.087239 30853 raft_consensus.cc:482] T b00a70de47d6453fa4864cf5d69157bd P 33e06028c7ef47f8ac35fcafb1372c71 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:00:57.087352 30853 raft_consensus.cc:504] T b00a70de47d6453fa4864cf5d69157bd P 33e06028c7ef47f8ac35fcafb1372c71 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33e06028c7ef47f8ac35fcafb1372c71" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 39559 } } peers { permanent_uuid: "b2154de8c41b4e83bac8daed7caa8973" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 37115 } } peers { permanent_uuid: "267c7a7418a74aa8b9e0828d24ed7709" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 40909 } }
I20260927 04:00:57.087685 30853 leader_election.cc:288] T b00a70de47d6453fa4864cf5d69157bd P 33e06028c7ef47f8ac35fcafb1372c71 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b2154de8c41b4e83bac8daed7caa8973 (127.15.185.65:37115), 267c7a7418a74aa8b9e0828d24ed7709 (127.15.185.66:40909)
I20260927 04:00:57.090794 30575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:57.088009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:53175 (local address 127.15.185.65:37115)
0927 04:00:57.088070 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 04:00:57.088073 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:00:57.088106 (+ 33us) server_negotiation.cc:407] Connection header received
0927 04:00:57.088141 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:00:57.088142 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:00:57.088178 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:00:57.088234 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:00:57.088426 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:57.088963 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:57.089983 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:57.090421 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:57.090517 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:57.090609 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 04:00:57.090654 (+ 45us) server_negotiation.cc:299] Negotiation successful
0927 04:00:57.090693 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 04:00:57.090781 30844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:57.087920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.65:37115 (local address 127.15.185.67:53175)
0927 04:00:57.087984 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:57.087998 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 04:00:57.088068 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:00:57.088264 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:00:57.088265 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:00:57.088267 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:00:57.088341 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:57.088342 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:57.089070 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:57.089071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:57.089878 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:57.089881 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:57.090515 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:57.090516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:57.090593 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:57.090597 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 04:00:57.090652 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 04:00:57.090696 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":49}
I20260927 04:00:57.090847 30712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:57.088066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:48999 (local address 127.15.185.66:40909)
0927 04:00:57.088110 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 04:00:57.088112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:00:57.088411 (+ 299us) server_negotiation.cc:407] Connection header received
0927 04:00:57.088436 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:00:57.088437 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:00:57.088446 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:00:57.088511 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:00:57.088657 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:57.089120 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:57.090070 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:57.090483 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:57.090573 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:57.090581 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 04:00:57.090732 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 04:00:57.090774 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 04:00:57.090934 30854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:57.087921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.66:40909 (local address 127.15.185.67:48999)
0927 04:00:57.088180 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:57.088188 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 04:00:57.088255 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:00:57.088513 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:00:57.088514 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:00:57.088515 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:00:57.088577 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:57.088578 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:57.089211 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:57.089213 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:57.089971 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:57.089973 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:57.090572 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:57.090573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:57.090653 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:57.090656 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 04:00:57.090726 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 04:00:57.090769 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"mutex_wait_us":85,"thread_start_us":93,"threads_started":1}
I20260927 04:00:57.091183 30521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b00a70de47d6453fa4864cf5d69157bd" candidate_uuid: "33e06028c7ef47f8ac35fcafb1372c71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2154de8c41b4e83bac8daed7caa8973" is_pre_election: true
I20260927 04:00:57.091396 30521 raft_consensus.cc:2454] T b00a70de47d6453fa4864cf5d69157bd P b2154de8c41b4e83bac8daed7caa8973 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33e06028c7ef47f8ac35fcafb1372c71 in term 0.
I20260927 04:00:57.091449 30660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b00a70de47d6453fa4864cf5d69157bd" candidate_uuid: "33e06028c7ef47f8ac35fcafb1372c71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "267c7a7418a74aa8b9e0828d24ed7709" is_pre_election: true
I20260927 04:00:57.091634 30660 raft_consensus.cc:2454] T b00a70de47d6453fa4864cf5d69157bd P 267c7a7418a74aa8b9e0828d24ed7709 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33e06028c7ef47f8ac35fcafb1372c71 in term 0.
I20260927 04:00:57.091660 30729 leader_election.cc:302] T b00a70de47d6453fa4864cf5d69157bd P 33e06028c7ef47f8ac35fcafb1372c71 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33e06028c7ef47f8ac35fcafb1372c71, b2154de8c41b4e83bac8daed7caa8973; no voters:
I20260927 04:00:57.091781 30853 raft_consensus.cc:2781] T b00a70de47d6453fa4864cf5d69157bd P 33e06028c7ef47f8ac35fcafb1372c71 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 04:00:57.091859 30853 raft_consensus.cc:482] T b00a70de47d6453fa4864cf5d69157bd P 33e06028c7ef47f8ac35fcafb1372c71 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 04:00:57.091888 30853 raft_consensus.cc:3037] T b00a70de47d6453fa4864cf5d69157bd P 33e06028c7ef47f8ac35fcafb1372c71 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:00:57.092602 30853 raft_consensus.cc:504] T b00a70de47d6453fa4864cf5d69157bd P 33e06028c7ef47f8ac35fcafb1372c71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33e06028c7ef47f8ac35fcafb1372c71" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 39559 } } peers { permanent_uuid: "b2154de8c41b4e83bac8daed7caa8973" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 37115 } } peers { permanent_uuid: "267c7a7418a74aa8b9e0828d24ed7709" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 40909 } }
I20260927 04:00:57.092746 30853 leader_election.cc:288] T b00a70de47d6453fa4864cf5d69157bd P 33e06028c7ef47f8ac35fcafb1372c71 [CANDIDATE]: Term 1 election: Requested vote from peers b2154de8c41b4e83bac8daed7caa8973 (127.15.185.65:37115), 267c7a7418a74aa8b9e0828d24ed7709 (127.15.185.66:40909)
I20260927 04:00:57.092948 30521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b00a70de47d6453fa4864cf5d69157bd" candidate_uuid: "33e06028c7ef47f8ac35fcafb1372c71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2154de8c41b4e83bac8daed7caa8973"
I20260927 04:00:57.092985 30660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b00a70de47d6453fa4864cf5d69157bd" candidate_uuid: "33e06028c7ef47f8ac35fcafb1372c71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "267c7a7418a74aa8b9e0828d24ed7709"
I20260927 04:00:57.093034 30521 raft_consensus.cc:3037] T b00a70de47d6453fa4864cf5d69157bd P b2154de8c41b4e83bac8daed7caa8973 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:00:57.093079 30660 raft_consensus.cc:3037] T b00a70de47d6453fa4864cf5d69157bd P 267c7a7418a74aa8b9e0828d24ed7709 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:00:57.093816 30521 raft_consensus.cc:2454] T b00a70de47d6453fa4864cf5d69157bd P b2154de8c41b4e83bac8daed7caa8973 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33e06028c7ef47f8ac35fcafb1372c71 in term 1.
I20260927 04:00:57.093953 30660 raft_consensus.cc:2454] T b00a70de47d6453fa4864cf5d69157bd P 267c7a7418a74aa8b9e0828d24ed7709 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33e06028c7ef47f8ac35fcafb1372c71 in term 1.
I20260927 04:00:57.093997 30729 leader_election.cc:302] T b00a70de47d6453fa4864cf5d69157bd P 33e06028c7ef47f8ac35fcafb1372c71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33e06028c7ef47f8ac35fcafb1372c71, b2154de8c41b4e83bac8daed7caa8973; no voters:
I20260927 04:00:57.094071 30853 raft_consensus.cc:2781] T b00a70de47d6453fa4864cf5d69157bd P 33e06028c7ef47f8ac35fcafb1372c71 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:00:57.094237 30853 raft_consensus.cc:686] T b00a70de47d6453fa4864cf5d69157bd P 33e06028c7ef47f8ac35fcafb1372c71 [term 1 LEADER]: Becoming Leader. State: Replica: 33e06028c7ef47f8ac35fcafb1372c71, State: Running, Role: LEADER
I20260927 04:00:57.094352 30853 consensus_queue.cc:237] T b00a70de47d6453fa4864cf5d69157bd P 33e06028c7ef47f8ac35fcafb1372c71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33e06028c7ef47f8ac35fcafb1372c71" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 39559 } } peers { permanent_uuid: "b2154de8c41b4e83bac8daed7caa8973" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 37115 } } peers { permanent_uuid: "267c7a7418a74aa8b9e0828d24ed7709" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 40909 } }
I20260927 04:00:57.095322 30378 catalog_manager.cc:5495] T b00a70de47d6453fa4864cf5d69157bd P 33e06028c7ef47f8ac35fcafb1372c71 reported cstate change: term changed from 0 to 1, leader changed from <none> to 33e06028c7ef47f8ac35fcafb1372c71 (127.15.185.67). New cstate: current_term: 1 leader_uuid: "33e06028c7ef47f8ac35fcafb1372c71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33e06028c7ef47f8ac35fcafb1372c71" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 39559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2154de8c41b4e83bac8daed7caa8973" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 37115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "267c7a7418a74aa8b9e0828d24ed7709" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 40909 } health_report { overall_health: UNKNOWN } } }
I20260927 04:00:57.104528 30848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:57.101925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53580 (local address 127.15.185.67:39559)
0927 04:00:57.102026 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 04:00:57.102028 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:00:57.102037 (+ 9us) server_negotiation.cc:407] Connection header received
0927 04:00:57.102069 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:00:57.102078 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:00:57.102096 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:00:57.102166 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:00:57.102262 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:57.102768 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:57.103549 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:57.103819 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:57.103889 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:57.104047 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:00:57.104165 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:00:57.104240 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 04:00:57.104273 (+ 33us) server_negotiation.cc:299] Negotiation successful
0927 04:00:57.104293 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 04:00:57.106484 30521 raft_consensus.cc:1261] T b00a70de47d6453fa4864cf5d69157bd P b2154de8c41b4e83bac8daed7caa8973 [term 1 FOLLOWER]: Refusing update from remote peer 33e06028c7ef47f8ac35fcafb1372c71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:00:57.106488 30660 raft_consensus.cc:1261] T b00a70de47d6453fa4864cf5d69157bd P 267c7a7418a74aa8b9e0828d24ed7709 [term 1 FOLLOWER]: Refusing update from remote peer 33e06028c7ef47f8ac35fcafb1372c71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:00:57.106861 30855 consensus_queue.cc:1035] T b00a70de47d6453fa4864cf5d69157bd P 33e06028c7ef47f8ac35fcafb1372c71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2154de8c41b4e83bac8daed7caa8973" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 37115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:00:57.107002 30853 consensus_queue.cc:1035] T b00a70de47d6453fa4864cf5d69157bd P 33e06028c7ef47f8ac35fcafb1372c71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "267c7a7418a74aa8b9e0828d24ed7709" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 40909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:00:57.109122 30858 mvcc.cc:205] Tried to move back new op lower bound from 7333812867505295360 to 7333812867459076096. Current Snapshot: MvccSnapshot[applied={T|T < 7333812867505295360}]
I20260927 04:00:57.109256 30859 mvcc.cc:205] Tried to move back new op lower bound from 7333812867505295360 to 7333812867459076096. Current Snapshot: MvccSnapshot[applied={T|T < 7333812867505295360}]
I20260927 04:00:57.109613 30860 mvcc.cc:205] Tried to move back new op lower bound from 7333812867505295360 to 7333812867459076096. Current Snapshot: MvccSnapshot[applied={T|T < 7333812867505295360}]
I20260927 04:00:57.111310 30378 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:42364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:00:57.111398 30378 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:42364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:00:57.112083 30378 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8c3e092adfae4d008e6c22c1c0c22a34: Sending DeleteTablet for 3 replicas of tablet b00a70de47d6453fa4864cf5d69157bd
I20260927 04:00:57.112352 30776 tablet_service.cc:1514] Processing DeleteTablet for tablet b00a70de47d6453fa4864cf5d69157bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:00:57 UTC) from {username='kudu', principal='kudu/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:53572
I20260927 04:00:57.112388 30640 tablet_service.cc:1514] Processing DeleteTablet for tablet b00a70de47d6453fa4864cf5d69157bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:00:57 UTC) from {username='kudu', principal='kudu/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:40274
I20260927 04:00:57.112423 30501 tablet_service.cc:1514] Processing DeleteTablet for tablet b00a70de47d6453fa4864cf5d69157bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:00:57 UTC) from {username='kudu', principal='kudu/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:49184
I20260927 04:00:57.112473 30713 tablet_replica.cc:330] T b00a70de47d6453fa4864cf5d69157bd P 267c7a7418a74aa8b9e0828d24ed7709: stopping tablet replica
I20260927 04:00:57.112520 30713 raft_consensus.cc:2229] T b00a70de47d6453fa4864cf5d69157bd P 267c7a7418a74aa8b9e0828d24ed7709 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:00:57.112540 30576 tablet_replica.cc:330] T b00a70de47d6453fa4864cf5d69157bd P b2154de8c41b4e83bac8daed7caa8973: stopping tablet replica
I20260927 04:00:57.112576 30868 tablet_replica.cc:330] T b00a70de47d6453fa4864cf5d69157bd P 33e06028c7ef47f8ac35fcafb1372c71: stopping tablet replica
I20260927 04:00:57.112588 30713 raft_consensus.cc:2258] T b00a70de47d6453fa4864cf5d69157bd P 267c7a7418a74aa8b9e0828d24ed7709 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:00:57.112605 30576 raft_consensus.cc:2229] T b00a70de47d6453fa4864cf5d69157bd P b2154de8c41b4e83bac8daed7caa8973 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:00:57.112670 30576 raft_consensus.cc:2258] T b00a70de47d6453fa4864cf5d69157bd P b2154de8c41b4e83bac8daed7caa8973 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:00:57.112673 30868 raft_consensus.cc:2229] T b00a70de47d6453fa4864cf5d69157bd P 33e06028c7ef47f8ac35fcafb1372c71 [term 1 LEADER]: Raft consensus shutting down.
I20260927 04:00:57.112790 30868 raft_consensus.cc:2258] T b00a70de47d6453fa4864cf5d69157bd P 33e06028c7ef47f8ac35fcafb1372c71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:00:57.112826 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 30418
I20260927 04:00:57.112998 30713 ts_tablet_manager.cc:1905] T b00a70de47d6453fa4864cf5d69157bd P 267c7a7418a74aa8b9e0828d24ed7709: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:00:57.112999 30576 ts_tablet_manager.cc:1905] T b00a70de47d6453fa4864cf5d69157bd P b2154de8c41b4e83bac8daed7caa8973: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:00:57.113802 30868 ts_tablet_manager.cc:1905] T b00a70de47d6453fa4864cf5d69157bd P 33e06028c7ef47f8ac35fcafb1372c71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:00:57.114681 30713 ts_tablet_manager.cc:1918] T b00a70de47d6453fa4864cf5d69157bd P 267c7a7418a74aa8b9e0828d24ed7709: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:00:57.114740 30713 log.cc:1196] T b00a70de47d6453fa4864cf5d69157bd P 267c7a7418a74aa8b9e0828d24ed7709: Deleting WAL directory at /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-1/wal/wals/b00a70de47d6453fa4864cf5d69157bd
I20260927 04:00:57.114974 30713 ts_tablet_manager.cc:1939] T b00a70de47d6453fa4864cf5d69157bd P 267c7a7418a74aa8b9e0828d24ed7709: Deleting consensus metadata
I20260927 04:00:57.115468 30366 catalog_manager.cc:4841] TS 267c7a7418a74aa8b9e0828d24ed7709 (127.15.185.66:40909): tablet b00a70de47d6453fa4864cf5d69157bd (table test-table [id=29cffce467df4fb79421ad074981b8d4]) successfully deleted
I20260927 04:00:57.115674 30868 ts_tablet_manager.cc:1918] T b00a70de47d6453fa4864cf5d69157bd P 33e06028c7ef47f8ac35fcafb1372c71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:00:57.115753 30868 log.cc:1196] T b00a70de47d6453fa4864cf5d69157bd P 33e06028c7ef47f8ac35fcafb1372c71: Deleting WAL directory at /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-2/wal/wals/b00a70de47d6453fa4864cf5d69157bd
I20260927 04:00:57.116060 30868 ts_tablet_manager.cc:1939] T b00a70de47d6453fa4864cf5d69157bd P 33e06028c7ef47f8ac35fcafb1372c71: Deleting consensus metadata
I20260927 04:00:57.116483 30364 catalog_manager.cc:4841] TS 33e06028c7ef47f8ac35fcafb1372c71 (127.15.185.67:39559): tablet b00a70de47d6453fa4864cf5d69157bd (table test-table [id=29cffce467df4fb79421ad074981b8d4]) successfully deleted
W20260927 04:00:57.120208 30363 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.185.65:37115 (error 108)
I20260927 04:00:57.120497 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 30571
I20260927 04:00:57.120782 30574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:57.120644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.65:37115 (local address 127.0.0.1:49192)
0927 04:00:57.120703 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:57.120747 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.65:37115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260927 04:00:57.121106 30363 catalog_manager.cc:4584] TS b2154de8c41b4e83bac8daed7caa8973 (127.15.185.65:37115): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b00a70de47d6453fa4864cf5d69157bd: Network error: Client connection negotiation failed: client connection to 127.15.185.65:37115: connect: Connection refused (error 111)
I20260927 04:00:57.125535 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 30709
I20260927 04:00:57.130573 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 30346
I20260927 04:00:57.135862 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:45853
--webserver_interface=127.15.185.126
--webserver_port=33989
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:39093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.185.126:45853
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:00:57.218338 30870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:00:57.218580 30870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:00:57.218621 30870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:00:57.219859 30870 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:00:57.219924 30870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:00:57.219956 30870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:00:57.219988 30870 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:00:57.220016 30870 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:00:57.221472 30870 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:39093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.185.126:45853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:45853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.185.126
--webserver_port=33989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:00:57.221752 30870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:00:57.221980 30870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:00:57.226794 30878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:00:57.226856 30875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:00:57.226794 30876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:00:57.226969 30870 server_base.cc:1034] running on GCE node
I20260927 04:00:57.227314 30870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:00:57.227628 30870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:00:57.228825 30870 hybrid_clock.cc:648] HybridClock initialized: now 1790481657228805 us; error 29 us; skew 500 ppm
Sep 27 04:00:57 dist-test-slave-glgz krb5kdc[29801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481657, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:00:57.231240 30870 init.cc:377] Logged in from keytab as kudu/127.15.185.126@KRBTEST.COM (short username kudu)
I20260927 04:00:57.231664 30870 webserver.cc:460] Webserver started at http://127.15.185.126:33989/ using document root <none> and password file <none>
I20260927 04:00:57.231900 30870 fs_manager.cc:359] Metadata directory not provided
I20260927 04:00:57.231967 30870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:00:57.233246 30870 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:00:57.233914 30885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:00:57.234092 30870 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 04:00:57.234187 30870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/master-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/master-0/wal
uuid: "8c3e092adfae4d008e6c22c1c0c22a34"
format_stamp: "Formatted at 2026-09-27 04:00:55 on dist-test-slave-glgz"
server_key: "828e675d78684da1271578c26259e0d5"
server_key_iv: "6c8a661f1af8a136fb39b7dc3eae775d"
server_key_version: "encryptionkey@0"
I20260927 04:00:57.234552 30870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:00:57.257891 30870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:00:57.259953 30870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:00:57.260123 30870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:00:57.264055 30870 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.126:45853
I20260927 04:00:57.264168 30937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.126:45853 every 8 connection(s)
I20260927 04:00:57.264518 30870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
I20260927 04:00:57.266059 30938 sys_catalog.cc:263] Verifying existing consensus state
I20260927 04:00:57.266654 30938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c3e092adfae4d008e6c22c1c0c22a34: Bootstrap starting.
I20260927 04:00:57.270705 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 30870
I20260927 04:00:57.271111 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.65:37115
--local_ip_for_outbound_sockets=127.15.185.65
--tserver_master_addrs=127.15.185.126:45853
--webserver_port=37765
--webserver_interface=127.15.185.65
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab
--principal=kudu/127.15.185.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:39093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 04:00:57.275158 30938 log.cc:824] T 00000000000000000000000000000000 P 8c3e092adfae4d008e6c22c1c0c22a34: Log is configured to *not* fsync() on all Append() calls
I20260927 04:00:57.277436 30938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c3e092adfae4d008e6c22c1c0c22a34: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 04:00:57.277784 30938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c3e092adfae4d008e6c22c1c0c22a34: Bootstrap complete.
I20260927 04:00:57.280184 30938 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c3e092adfae4d008e6c22c1c0c22a34 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c3e092adfae4d008e6c22c1c0c22a34" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45853 } }
I20260927 04:00:57.280413 30938 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c3e092adfae4d008e6c22c1c0c22a34 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c3e092adfae4d008e6c22c1c0c22a34, State: Initialized, Role: FOLLOWER
I20260927 04:00:57.280587 30938 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c3e092adfae4d008e6c22c1c0c22a34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c3e092adfae4d008e6c22c1c0c22a34" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45853 } }
I20260927 04:00:57.280663 30938 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c3e092adfae4d008e6c22c1c0c22a34 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:00:57.280719 30938 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c3e092adfae4d008e6c22c1c0c22a34 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:00:57.280771 30938 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c3e092adfae4d008e6c22c1c0c22a34 [term 2 FOLLOWER]: Advancing to term 3
I20260927 04:00:57.281839 30938 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c3e092adfae4d008e6c22c1c0c22a34 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c3e092adfae4d008e6c22c1c0c22a34" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45853 } }
I20260927 04:00:57.282068 30938 leader_election.cc:302] T 00000000000000000000000000000000 P 8c3e092adfae4d008e6c22c1c0c22a34 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c3e092adfae4d008e6c22c1c0c22a34; no voters:
I20260927 04:00:57.282321 30938 leader_election.cc:288] T 00000000000000000000000000000000 P 8c3e092adfae4d008e6c22c1c0c22a34 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 04:00:57.282358 30942 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c3e092adfae4d008e6c22c1c0c22a34 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 04:00:57.282464 30942 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c3e092adfae4d008e6c22c1c0c22a34 [term 3 LEADER]: Becoming Leader. State: Replica: 8c3e092adfae4d008e6c22c1c0c22a34, State: Running, Role: LEADER
I20260927 04:00:57.282565 30942 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c3e092adfae4d008e6c22c1c0c22a34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c3e092adfae4d008e6c22c1c0c22a34" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45853 } }
I20260927 04:00:57.282612 30938 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c3e092adfae4d008e6c22c1c0c22a34 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:00:57.282828 30942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c3e092adfae4d008e6c22c1c0c22a34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "8c3e092adfae4d008e6c22c1c0c22a34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c3e092adfae4d008e6c22c1c0c22a34" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45853 } } }
I20260927 04:00:57.282895 30942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c3e092adfae4d008e6c22c1c0c22a34 [sys.catalog]: This master's current role is: LEADER
I20260927 04:00:57.283011 30942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c3e092adfae4d008e6c22c1c0c22a34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c3e092adfae4d008e6c22c1c0c22a34. Latest consensus state: current_term: 3 leader_uuid: "8c3e092adfae4d008e6c22c1c0c22a34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c3e092adfae4d008e6c22c1c0c22a34" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45853 } } }
I20260927 04:00:57.283107 30942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c3e092adfae4d008e6c22c1c0c22a34 [sys.catalog]: This master's current role is: LEADER
I20260927 04:00:57.283422 30946 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:00:57.284389 30946 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:00:57.284492 30946 catalog_manager.cc:1259] Loaded cluster ID: 4ef75bee74c94e119a74adff394a9774
I20260927 04:00:57.284518 30946 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260927 04:00:57.284749 30946 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fd39bdf79ad google::LogMessage::Fail() at ??:0
@ 0x7fd39bdfa88a google::LogMessage::SendToLog() at ??:0
@ 0x7fd39bdf74de google::LogMessage::Flush() at ??:0
@ 0x7fd39bdf8a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fd3a3d1c0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fd3a3d258b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fd39dcc50d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fd39dcba954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fd39eb1e6db start_thread at ??:0
@ 0x7fd39a8f561f clone at ??:0
W20260927 04:00:57.371424 30940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:00:57.371620 30940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:00:57.371644 30940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:00:57.372915 30940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:00:57.372963 30940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:00:57.372991 30940 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.65
I20260927 04:00:57.374495 30940 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:39093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab
--principal=kudu/127.15.185.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.65:37115
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.185.65
--webserver_port=37765
--tserver_master_addrs=127.15.185.126:45853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.65
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:00:57.374773 30940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:00:57.375017 30940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:00:57.380437 30974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:00:57.380511 30977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:00:57.380422 30975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:00:57.380796 30940 server_base.cc:1034] running on GCE node
I20260927 04:00:57.381029 30940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:00:57.381312 30940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:00:57.382491 30940 hybrid_clock.cc:648] HybridClock initialized: now 1790481657382469 us; error 32 us; skew 500 ppm
Sep 27 04:00:57 dist-test-slave-glgz krb5kdc[29801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481657, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:00:57.385589 30940 init.cc:377] Logged in from keytab as kudu/127.15.185.65@KRBTEST.COM (short username kudu)
I20260927 04:00:57.386255 30940 webserver.cc:460] Webserver started at http://127.15.185.65:37765/ using document root <none> and password file <none>
I20260927 04:00:57.386555 30940 fs_manager.cc:359] Metadata directory not provided
I20260927 04:00:57.386633 30940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:00:57.388391 30940 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:00:57.389269 30984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:00:57.389480 30940 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:00:57.389580 30940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-0/wal
uuid: "b2154de8c41b4e83bac8daed7caa8973"
format_stamp: "Formatted at 2026-09-27 04:00:55 on dist-test-slave-glgz"
server_key: "97f2390d35e4e67b41727c7fe49bcd58"
server_key_iv: "1ce6a91b6c7c9d179923f02c0a739430"
server_key_version: "encryptionkey@0"
I20260927 04:00:57.390071 30940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:00:57.415000 30940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:00:57.417899 30940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:00:57.418075 30940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:00:57.418376 30940 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:00:57.418946 30991 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 04:00:57.420487 30940 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 04:00:57.420554 30940 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 04:00:57.420589 30940 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 04:00:57.421463 30940 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 04:00:57.421510 30940 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 04:00:57.421654 30991 tablet_bootstrap.cc:492] T b00a70de47d6453fa4864cf5d69157bd P b2154de8c41b4e83bac8daed7caa8973: Bootstrap starting.
I20260927 04:00:57.423735 30991 log.cc:824] T b00a70de47d6453fa4864cf5d69157bd P b2154de8c41b4e83bac8daed7caa8973: Log is configured to *not* fsync() on all Append() calls
I20260927 04:00:57.427459 30991 tablet_bootstrap.cc:492] T b00a70de47d6453fa4864cf5d69157bd P b2154de8c41b4e83bac8daed7caa8973: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 04:00:57.428174 30991 tablet_bootstrap.cc:492] T b00a70de47d6453fa4864cf5d69157bd P b2154de8c41b4e83bac8daed7caa8973: Bootstrap complete.
I20260927 04:00:57.428375 30991 ts_tablet_manager.cc:1397] T b00a70de47d6453fa4864cf5d69157bd P b2154de8c41b4e83bac8daed7caa8973: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260927 04:00:57.428941 30940 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.65:37115
I20260927 04:00:57.429103 31099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.65:37115 every 8 connection(s)
I20260927 04:00:57.429484 30940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
I20260927 04:00:57.430359 30991 raft_consensus.cc:348] T b00a70de47d6453fa4864cf5d69157bd P b2154de8c41b4e83bac8daed7caa8973 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33e06028c7ef47f8ac35fcafb1372c71" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 39559 } } peers { permanent_uuid: "b2154de8c41b4e83bac8daed7caa8973" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 37115 } } peers { permanent_uuid: "267c7a7418a74aa8b9e0828d24ed7709" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 40909 } }
I20260927 04:00:57.430660 30991 raft_consensus.cc:729] T b00a70de47d6453fa4864cf5d69157bd P b2154de8c41b4e83bac8daed7caa8973 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2154de8c41b4e83bac8daed7caa8973, State: Initialized, Role: FOLLOWER
I20260927 04:00:57.430824 30991 consensus_queue.cc:260] T b00a70de47d6453fa4864cf5d69157bd P b2154de8c41b4e83bac8daed7caa8973 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33e06028c7ef47f8ac35fcafb1372c71" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 39559 } } peers { permanent_uuid: "b2154de8c41b4e83bac8daed7caa8973" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 37115 } } peers { permanent_uuid: "267c7a7418a74aa8b9e0828d24ed7709" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 40909 } }
I20260927 04:00:57.431139 30991 ts_tablet_manager.cc:1428] T b00a70de47d6453fa4864cf5d69157bd P b2154de8c41b4e83bac8daed7caa8973: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 04:00:57.436537 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 30940
I20260927 04:00:57.436813 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.66:40909
--local_ip_for_outbound_sockets=127.15.185.66
--tserver_master_addrs=127.15.185.126:45853
--webserver_port=33105
--webserver_interface=127.15.185.66
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab
--principal=kudu/127.15.185.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:39093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:00:57.478570 31103 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:00:57.431354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45853 (local address 127.15.185.65:59343)
0927 04:00:57.431529 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:57.431565 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 04:00:57.431786 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:00:57.478331 (+ 46545us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:45853: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":77,"threads_started":1}
W20260927 04:00:57.478863 31100 heartbeater.cc:644] Failed to heartbeat to 127.15.185.126:45853 (0 consecutive failures): Network error: Failed to ping master at 127.15.185.126:45853: Client connection negotiation failed: client connection to 127.15.185.126:45853: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 04:00:57.479269 31103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:57.479145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45853 (local address 127.15.185.65:54473)
0927 04:00:57.479199 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:57.479238 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 04:00:57.479637 31103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:57.479553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45853 (local address 127.15.185.65:40671)
0927 04:00:57.479600 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:57.479622 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260927 04:00:57.479763 31100 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 04:00:57.527536 31104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:00:57.527776 31104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:00:57.527802 31104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:00:57.529363 31104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:00:57.529424 31104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:00:57.529479 31104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.66
I20260927 04:00:57.531142 31104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:39093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab
--principal=kudu/127.15.185.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.66:40909
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.185.66
--webserver_port=33105
--tserver_master_addrs=127.15.185.126:45853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.66
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:00:57.531419 31104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:00:57.531703 31104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:00:57.537146 31109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:00:57.537151 31112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:00:57.537171 31110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:00:57.537436 31104 server_base.cc:1034] running on GCE node
I20260927 04:00:57.537657 31104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:00:57.537915 31104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:00:57.539145 31104 hybrid_clock.cc:648] HybridClock initialized: now 1790481657539115 us; error 35 us; skew 500 ppm
Sep 27 04:00:57 dist-test-slave-glgz krb5kdc[29801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481657, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:00:57.541786 31104 init.cc:377] Logged in from keytab as kudu/127.15.185.66@KRBTEST.COM (short username kudu)
I20260927 04:00:57.542351 31104 webserver.cc:460] Webserver started at http://127.15.185.66:33105/ using document root <none> and password file <none>
I20260927 04:00:57.542584 31104 fs_manager.cc:359] Metadata directory not provided
I20260927 04:00:57.542654 31104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:00:57.544059 31104 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:00:57.544780 31119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:00:57.544951 31104 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:00:57.545056 31104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-1/wal
uuid: "267c7a7418a74aa8b9e0828d24ed7709"
format_stamp: "Formatted at 2026-09-27 04:00:56 on dist-test-slave-glgz"
server_key: "686c6646d3e856802f2dc7ec5e09dfeb"
server_key_iv: "86d776cdce8de99266ca638d4cc57002"
server_key_version: "encryptionkey@0"
I20260927 04:00:57.545423 31104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:00:57.553159 31104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:00:57.555119 31104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:00:57.555275 31104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:00:57.555495 31104 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:00:57.555799 31104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:00:57.555857 31104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:00:57.555902 31104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:00:57.555927 31104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:00:57.561898 31104 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.66:40909
I20260927 04:00:57.561939 31232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.66:40909 every 8 connection(s)
I20260927 04:00:57.562273 31104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
I20260927 04:00:57.563315 31235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:57.562897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45853 (local address 127.15.185.66:54371)
0927 04:00:57.563082 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:57.563180 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":66,"threads_started":1}
W20260927 04:00:57.563727 31233 heartbeater.cc:644] Failed to heartbeat to 127.15.185.126:45853 (0 consecutive failures): Network error: Failed to ping master at 127.15.185.126:45853: Client connection negotiation failed: client connection to 127.15.185.126:45853: connect: Connection refused (error 111)
I20260927 04:00:57.564019 31235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:57.563931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45853 (local address 127.15.185.66:41213)
0927 04:00:57.563975 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:57.564002 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 04:00:57.564313 31235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:57.564244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45853 (local address 127.15.185.66:36771)
0927 04:00:57.564280 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:57.564300 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260927 04:00:57.564460 31233 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 04:00:57.572495 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 31104
I20260927 04:00:57.572769 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.67:39559
--local_ip_for_outbound_sockets=127.15.185.67
--tserver_master_addrs=127.15.185.126:45853
--webserver_port=42267
--webserver_interface=127.15.185.67
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab
--principal=kudu/127.15.185.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:39093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:00:57.655026 31236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:00:57.655272 31236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:00:57.655296 31236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:00:57.656482 31236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:00:57.656523 31236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:00:57.656549 31236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.67
I20260927 04:00:57.658094 31236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:39093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab
--principal=kudu/127.15.185.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.67:39559
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.185.67
--webserver_port=42267
--tserver_master_addrs=127.15.185.126:45853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.67
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:00:57.658363 31236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:00:57.658605 31236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:00:57.663560 31241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:00:57.663606 31242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:00:57.663780 31236 server_base.cc:1034] running on GCE node
W20260927 04:00:57.663869 31244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:00:57.664096 31236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:00:57.664325 31236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:00:57.665493 31236 hybrid_clock.cc:648] HybridClock initialized: now 1790481657665492 us; error 37 us; skew 500 ppm
Sep 27 04:00:57 dist-test-slave-glgz krb5kdc[29801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481657, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:00:57.668118 31236 init.cc:377] Logged in from keytab as kudu/127.15.185.67@KRBTEST.COM (short username kudu)
I20260927 04:00:57.668577 31236 webserver.cc:460] Webserver started at http://127.15.185.67:42267/ using document root <none> and password file <none>
I20260927 04:00:57.668818 31236 fs_manager.cc:359] Metadata directory not provided
I20260927 04:00:57.668891 31236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:00:57.670215 31236 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:00:57.670852 31251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:00:57.671036 31236 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:00:57.671180 31236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-2/wal
uuid: "33e06028c7ef47f8ac35fcafb1372c71"
format_stamp: "Formatted at 2026-09-27 04:00:56 on dist-test-slave-glgz"
server_key: "630f3b08295e9dabe1c90c06649bc480"
server_key_iv: "45015d192198317bba13fdfb3131306e"
server_key_version: "encryptionkey@0"
I20260927 04:00:57.671545 31236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:00:57.680012 31236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:00:57.682343 31236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:00:57.682461 31236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:00:57.682668 31236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:00:57.682955 31236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:00:57.682988 31236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:00:57.683010 31236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:00:57.683024 31236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:00:57.689002 31236 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.67:39559
I20260927 04:00:57.689088 31364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.67:39559 every 8 connection(s)
I20260927 04:00:57.689445 31236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
I20260927 04:00:57.690549 31367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:57.690161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45853 (local address 127.15.185.67:60239)
0927 04:00:57.690318 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:57.690405 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":74,"threads_started":1}
W20260927 04:00:57.690941 31365 heartbeater.cc:644] Failed to heartbeat to 127.15.185.126:45853 (0 consecutive failures): Network error: Failed to ping master at 127.15.185.126:45853: Client connection negotiation failed: client connection to 127.15.185.126:45853: connect: Connection refused (error 111)
I20260927 04:00:57.691208 31367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:57.691138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45853 (local address 127.15.185.67:44821)
0927 04:00:57.691166 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:57.691190 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
I20260927 04:00:57.691454 31367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:57.691401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45853 (local address 127.15.185.67:52525)
0927 04:00:57.691420 (+ 19us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:57.691438 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260927 04:00:57.691555 31365 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 04:00:57.697839 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 31236
I20260927 04:00:58.480407 31369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:58.480147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45853 (local address 127.15.185.65:41433)
0927 04:00:58.480310 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:58.480348 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":112,"threads_started":1}
I20260927 04:00:58.565014 31370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:58.564776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45853 (local address 127.15.185.66:54669)
0927 04:00:58.564892 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:58.564929 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":82,"threads_started":1}
I20260927 04:00:58.692080 31371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:58.691842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45853 (local address 127.15.185.67:41987)
0927 04:00:58.691975 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:58.692022 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":87,"threads_started":1}
I20260927 04:00:58.970013 31372 raft_consensus.cc:482] T b00a70de47d6453fa4864cf5d69157bd P b2154de8c41b4e83bac8daed7caa8973 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:00:58.970141 31372 raft_consensus.cc:504] T b00a70de47d6453fa4864cf5d69157bd P b2154de8c41b4e83bac8daed7caa8973 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33e06028c7ef47f8ac35fcafb1372c71" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 39559 } } peers { permanent_uuid: "b2154de8c41b4e83bac8daed7caa8973" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 37115 } } peers { permanent_uuid: "267c7a7418a74aa8b9e0828d24ed7709" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 40909 } }
I20260927 04:00:58.970438 31372 leader_election.cc:288] T b00a70de47d6453fa4864cf5d69157bd P b2154de8c41b4e83bac8daed7caa8973 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 33e06028c7ef47f8ac35fcafb1372c71 (127.15.185.67:39559), 267c7a7418a74aa8b9e0828d24ed7709 (127.15.185.66:40909)
Sep 27 04:00:58 dist-test-slave-glgz krb5kdc[29801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481657, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.65@KRBTEST.COM for kudu/127.15.185.67@KRBTEST.COM
Sep 27 04:00:58 dist-test-slave-glgz krb5kdc[29801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481657, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.65@KRBTEST.COM for kudu/127.15.185.66@KRBTEST.COM
I20260927 04:00:58.981287 31374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:58.970772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:45553 (local address 127.15.185.66:40909)
0927 04:00:58.970920 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 04:00:58.970939 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 04:00:58.970965 (+ 26us) server_negotiation.cc:407] Connection header received
0927 04:00:58.971468 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:00:58.971470 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:00:58.971514 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:00:58.971627 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:00:58.972692 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:58.973572 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:58.974594 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:58.975216 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:58.975348 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:58.978027 (+ 2679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:00:58.978031 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:00:58.978031 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:00:58.978043 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:00:58.979658 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:58.980086 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:58.980091 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:58.980092 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:58.980179 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:58.980431 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:58.980432 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:58.980432 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:58.980662 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:00:58.980747 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 04:00:58.981043 (+ 296us) server_negotiation.cc:299] Negotiation successful
0927 04:00:58.981168 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":58,"threads_started":1}
I20260927 04:00:58.981287 31369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:58.970641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.66:40909 (local address 127.15.185.65:45553)
0927 04:00:58.970706 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:58.970720 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 04:00:58.970793 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:00:58.971701 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:00:58.971705 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:00:58.972102 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:00:58.972582 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:58.972584 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:58.973693 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:58.973694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:58.974502 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:58.974505 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:58.975309 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:58.975310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:58.975381 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:58.976018 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:00:58.976022 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:00:58.977921 (+ 1899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:00:58.979748 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:00:58.979749 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:00:58.979750 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:00:58.979997 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:00:58.980254 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:00:58.980255 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:00:58.980256 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:00:58.980362 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:00:58.980740 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:00:58.980747 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:00:58.980885 (+ 138us) client_negotiation.cc:769] Sending connection context
0927 04:00:58.980993 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 04:00:58.981180 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":55}
I20260927 04:00:58.981300 31375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:58.970882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:44257 (local address 127.15.185.67:39559)
0927 04:00:58.971012 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 04:00:58.971020 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 04:00:58.971066 (+ 46us) server_negotiation.cc:407] Connection header received
0927 04:00:58.971500 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:00:58.971502 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:00:58.971547 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:00:58.971708 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:00:58.972466 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:58.973431 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:58.974628 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:00:58.975102 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:00:58.975227 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:58.977703 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:00:58.977707 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:00:58.977707 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:00:58.977723 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:00:58.979587 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:58.980063 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:58.980067 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:58.980067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:58.980160 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:00:58.980424 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:00:58.980425 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:00:58.980426 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:00:58.980662 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:00:58.980762 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 04:00:58.981044 (+ 282us) server_negotiation.cc:299] Negotiation successful
0927 04:00:58.981168 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":58,"threads_started":1}
I20260927 04:00:58.981418 31373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:58.970640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.67:39559 (local address 127.15.185.65:44257)
0927 04:00:58.970759 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:58.970770 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 04:00:58.970830 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:00:58.971713 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:00:58.971714 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:00:58.972104 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:00:58.972325 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:58.972332 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:58.973536 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:58.973538 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:58.974502 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:00:58.974504 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:00:58.975216 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:00:58.975225 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:00:58.975347 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:00:58.975950 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:00:58.975958 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:00:58.977543 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:00:58.979692 (+ 2149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:00:58.979699 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:00:58.979701 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:00:58.979967 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:00:58.980244 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:00:58.980245 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:00:58.980245 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:00:58.980357 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:00:58.980759 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:00:58.980760 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:00:58.980885 (+ 125us) client_negotiation.cc:769] Sending connection context
0927 04:00:58.980993 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 04:00:58.981268 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":189,"thread_start_us":46,"threads_started":1}
W20260927 04:00:58.982060 30988 leader_election.cc:341] T b00a70de47d6453fa4864cf5d69157bd P b2154de8c41b4e83bac8daed7caa8973 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 267c7a7418a74aa8b9e0828d24ed7709 (127.15.185.66:40909): Not found: Tablet not found: b00a70de47d6453fa4864cf5d69157bd
W20260927 04:00:58.982544 30986 leader_election.cc:341] T b00a70de47d6453fa4864cf5d69157bd P b2154de8c41b4e83bac8daed7caa8973 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 33e06028c7ef47f8ac35fcafb1372c71 (127.15.185.67:39559): Not found: Tablet not found: b00a70de47d6453fa4864cf5d69157bd
I20260927 04:00:58.982613 30986 leader_election.cc:302] T b00a70de47d6453fa4864cf5d69157bd P b2154de8c41b4e83bac8daed7caa8973 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b2154de8c41b4e83bac8daed7caa8973; no voters: 267c7a7418a74aa8b9e0828d24ed7709, 33e06028c7ef47f8ac35fcafb1372c71
I20260927 04:00:58.982750 31372 raft_consensus.cc:2726] T b00a70de47d6453fa4864cf5d69157bd P b2154de8c41b4e83bac8daed7caa8973 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 04:00:59.481052 31373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:59.480890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45853 (local address 127.15.185.65:39861)
0927 04:00:59.480970 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:59.481012 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 04:00:59.565593 31376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:59.565407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45853 (local address 127.15.185.66:32869)
0927 04:00:59.565521 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:59.565556 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":56,"threads_started":1}
I20260927 04:00:59.692787 31377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:00:59.692523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45853 (local address 127.15.185.67:38667)
0927 04:00:59.692650 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 04:00:59.692742 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":55,"threads_started":1}
I20260927 04:01:00.481822 31378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:00.481605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45853 (local address 127.15.185.65:44697)
0927 04:01:00.481742 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:00.481775 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":80,"threads_started":1}
I20260927 04:01:00.566246 31379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:00.566046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45853 (local address 127.15.185.66:60487)
0927 04:01:00.566167 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:00.566205 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":57,"threads_started":1}
I20260927 04:01:00.693399 31380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:00.693179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45853 (local address 127.15.185.67:49053)
0927 04:01:00.693316 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:00.693356 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":58,"threads_started":1}
I20260927 04:01:01.130012 31381 raft_consensus.cc:482] T b00a70de47d6453fa4864cf5d69157bd P b2154de8c41b4e83bac8daed7caa8973 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:01:01.130115 31381 raft_consensus.cc:504] T b00a70de47d6453fa4864cf5d69157bd P b2154de8c41b4e83bac8daed7caa8973 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33e06028c7ef47f8ac35fcafb1372c71" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 39559 } } peers { permanent_uuid: "b2154de8c41b4e83bac8daed7caa8973" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 37115 } } peers { permanent_uuid: "267c7a7418a74aa8b9e0828d24ed7709" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 40909 } }
I20260927 04:01:01.130290 31381 leader_election.cc:288] T b00a70de47d6453fa4864cf5d69157bd P b2154de8c41b4e83bac8daed7caa8973 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 33e06028c7ef47f8ac35fcafb1372c71 (127.15.185.67:39559), 267c7a7418a74aa8b9e0828d24ed7709 (127.15.185.66:40909)
W20260927 04:01:01.130717 30988 leader_election.cc:341] T b00a70de47d6453fa4864cf5d69157bd P b2154de8c41b4e83bac8daed7caa8973 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 267c7a7418a74aa8b9e0828d24ed7709 (127.15.185.66:40909): Not found: Tablet not found: b00a70de47d6453fa4864cf5d69157bd
W20260927 04:01:01.130803 30986 leader_election.cc:341] T b00a70de47d6453fa4864cf5d69157bd P b2154de8c41b4e83bac8daed7caa8973 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 33e06028c7ef47f8ac35fcafb1372c71 (127.15.185.67:39559): Not found: Tablet not found: b00a70de47d6453fa4864cf5d69157bd
I20260927 04:01:01.130887 30986 leader_election.cc:302] T b00a70de47d6453fa4864cf5d69157bd P b2154de8c41b4e83bac8daed7caa8973 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b2154de8c41b4e83bac8daed7caa8973; no voters: 267c7a7418a74aa8b9e0828d24ed7709, 33e06028c7ef47f8ac35fcafb1372c71
I20260927 04:01:01.130985 31381 raft_consensus.cc:2726] T b00a70de47d6453fa4864cf5d69157bd P b2154de8c41b4e83bac8daed7caa8973 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 04:01:01.482550 31382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:01.482327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45853 (local address 127.15.185.65:37589)
0927 04:01:01.482459 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:01.482503 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":57,"threads_started":1}
I20260927 04:01:01.566937 31383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:01.566713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45853 (local address 127.15.185.66:60063)
0927 04:01:01.566848 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:01.566891 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":61,"threads_started":1}
I20260927 04:01:01.694079 31384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:01.693886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45853 (local address 127.15.185.67:35763)
0927 04:01:01.694000 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:01.694039 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":57,"threads_started":1}
I20260927 04:01:02.483439 31385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:02.483183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45853 (local address 127.15.185.65:37109)
0927 04:01:02.483344 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:02.483391 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":65,"threads_started":1}
I20260927 04:01:02.567672 31386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:02.567411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45853 (local address 127.15.185.66:37283)
0927 04:01:02.567577 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:02.567620 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":59,"threads_started":1}
I20260927 04:01:02.694854 31387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:02.694588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45853 (local address 127.15.185.67:43051)
0927 04:01:02.694718 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:02.694797 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":60,"threads_started":1}
I20260927 04:01:02.982895 31388 raft_consensus.cc:482] T b00a70de47d6453fa4864cf5d69157bd P b2154de8c41b4e83bac8daed7caa8973 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:01:02.982983 31388 raft_consensus.cc:504] T b00a70de47d6453fa4864cf5d69157bd P b2154de8c41b4e83bac8daed7caa8973 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33e06028c7ef47f8ac35fcafb1372c71" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 39559 } } peers { permanent_uuid: "b2154de8c41b4e83bac8daed7caa8973" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 37115 } } peers { permanent_uuid: "267c7a7418a74aa8b9e0828d24ed7709" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 40909 } }
I20260927 04:01:02.983214 31388 leader_election.cc:288] T b00a70de47d6453fa4864cf5d69157bd P b2154de8c41b4e83bac8daed7caa8973 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 33e06028c7ef47f8ac35fcafb1372c71 (127.15.185.67:39559), 267c7a7418a74aa8b9e0828d24ed7709 (127.15.185.66:40909)
W20260927 04:01:02.983587 30986 leader_election.cc:341] T b00a70de47d6453fa4864cf5d69157bd P b2154de8c41b4e83bac8daed7caa8973 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 33e06028c7ef47f8ac35fcafb1372c71 (127.15.185.67:39559): Not found: Tablet not found: b00a70de47d6453fa4864cf5d69157bd
W20260927 04:01:02.983664 30988 leader_election.cc:341] T b00a70de47d6453fa4864cf5d69157bd P b2154de8c41b4e83bac8daed7caa8973 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 267c7a7418a74aa8b9e0828d24ed7709 (127.15.185.66:40909): Not found: Tablet not found: b00a70de47d6453fa4864cf5d69157bd
I20260927 04:01:02.983726 30988 leader_election.cc:302] T b00a70de47d6453fa4864cf5d69157bd P b2154de8c41b4e83bac8daed7caa8973 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b2154de8c41b4e83bac8daed7caa8973; no voters: 267c7a7418a74aa8b9e0828d24ed7709, 33e06028c7ef47f8ac35fcafb1372c71
I20260927 04:01:02.983806 31388 raft_consensus.cc:2726] T b00a70de47d6453fa4864cf5d69157bd P b2154de8c41b4e83bac8daed7caa8973 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 04:01:03.484179 31389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:03.483934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45853 (local address 127.15.185.65:58955)
0927 04:01:03.484081 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:03.484125 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":60,"threads_started":1}
I20260927 04:01:03.568435 31390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:03.568201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45853 (local address 127.15.185.66:52385)
0927 04:01:03.568350 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:03.568391 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":61,"threads_started":1}
I20260927 04:01:03.695518 31391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:03.695295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45853 (local address 127.15.185.67:43151)
0927 04:01:03.695433 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:03.695472 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":61,"threads_started":1}
I20260927 04:01:04.485064 31392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:04.484810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45853 (local address 127.15.185.65:49663)
0927 04:01:04.484970 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:04.485017 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":71,"threads_started":1}
I20260927 04:01:04.569264 31393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:04.569004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45853 (local address 127.15.185.66:36345)
0927 04:01:04.569164 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:04.569216 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":77,"threads_started":1}
I20260927 04:01:04.696224 31394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:04.696004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45853 (local address 127.15.185.67:40679)
0927 04:01:04.696139 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:04.696180 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":59,"threads_started":1}
I20260927 04:01:05.485996 31395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:05.485718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45853 (local address 127.15.185.65:58353)
0927 04:01:05.485899 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:05.485950 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":87,"threads_started":1}
I20260927 04:01:05.570111 31396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:05.569862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45853 (local address 127.15.185.66:39503)
0927 04:01:05.570015 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:05.570063 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":64,"threads_started":1}
I20260927 04:01:05.697026 31397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:05.696741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45853 (local address 127.15.185.67:45491)
0927 04:01:05.696931 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:05.696979 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":78,"threads_started":1}
I20260927 04:01:06.486706 31398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:06.486489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45853 (local address 127.15.185.65:33055)
0927 04:01:06.486607 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:06.486650 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":54,"threads_started":1}
I20260927 04:01:06.570936 31399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:06.570689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45853 (local address 127.15.185.66:49579)
0927 04:01:06.570842 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:06.570889 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":68,"threads_started":1}
I20260927 04:01:06.697834 31400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:06.697593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45853 (local address 127.15.185.67:33937)
0927 04:01:06.697738 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:06.697786 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":62,"threads_started":1}
I20260927 04:01:07.487695 31401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:07.487443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45853 (local address 127.15.185.65:46773)
0927 04:01:07.487591 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:07.487647 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":67,"threads_started":1}
I20260927 04:01:07.571777 31402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:07.571549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45853 (local address 127.15.185.66:39989)
0927 04:01:07.571687 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:07.571730 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":77,"threads_started":1}
I20260927 04:01:07.640586 31403 raft_consensus.cc:482] T b00a70de47d6453fa4864cf5d69157bd P b2154de8c41b4e83bac8daed7caa8973 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:01:07.640734 31403 raft_consensus.cc:504] T b00a70de47d6453fa4864cf5d69157bd P b2154de8c41b4e83bac8daed7caa8973 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33e06028c7ef47f8ac35fcafb1372c71" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 39559 } } peers { permanent_uuid: "b2154de8c41b4e83bac8daed7caa8973" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 37115 } } peers { permanent_uuid: "267c7a7418a74aa8b9e0828d24ed7709" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 40909 } }
I20260927 04:01:07.640954 31403 leader_election.cc:288] T b00a70de47d6453fa4864cf5d69157bd P b2154de8c41b4e83bac8daed7caa8973 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 33e06028c7ef47f8ac35fcafb1372c71 (127.15.185.67:39559), 267c7a7418a74aa8b9e0828d24ed7709 (127.15.185.66:40909)
W20260927 04:01:07.641402 30986 leader_election.cc:341] T b00a70de47d6453fa4864cf5d69157bd P b2154de8c41b4e83bac8daed7caa8973 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 33e06028c7ef47f8ac35fcafb1372c71 (127.15.185.67:39559): Not found: Tablet not found: b00a70de47d6453fa4864cf5d69157bd
W20260927 04:01:07.641499 30988 leader_election.cc:341] T b00a70de47d6453fa4864cf5d69157bd P b2154de8c41b4e83bac8daed7caa8973 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 267c7a7418a74aa8b9e0828d24ed7709 (127.15.185.66:40909): Not found: Tablet not found: b00a70de47d6453fa4864cf5d69157bd
I20260927 04:01:07.641563 30988 leader_election.cc:302] T b00a70de47d6453fa4864cf5d69157bd P b2154de8c41b4e83bac8daed7caa8973 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b2154de8c41b4e83bac8daed7caa8973; no voters: 267c7a7418a74aa8b9e0828d24ed7709, 33e06028c7ef47f8ac35fcafb1372c71
I20260927 04:01:07.641656 31403 raft_consensus.cc:2726] T b00a70de47d6453fa4864cf5d69157bd P b2154de8c41b4e83bac8daed7caa8973 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 04:01:07.698626 31404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:07.698380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45853 (local address 127.15.185.67:45291)
0927 04:01:07.698535 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:07.698577 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":81,"threads_started":1}
I20260927 04:01:08.488407 31405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:08.488191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45853 (local address 127.15.185.65:40739)
0927 04:01:08.488324 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:08.488359 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":79,"threads_started":1}
I20260927 04:01:08.572471 31406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:08.572226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45853 (local address 127.15.185.66:53347)
0927 04:01:08.572368 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:08.572409 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":62,"threads_started":1}
I20260927 04:01:08.699395 31407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:08.699165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45853 (local address 127.15.185.67:49061)
0927 04:01:08.699310 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:08.699349 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":60,"threads_started":1}
I20260927 04:01:09.489233 31408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:09.488950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45853 (local address 127.15.185.65:49171)
0927 04:01:09.489137 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:09.489184 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":79,"threads_started":1}
I20260927 04:01:09.573318 31409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:09.573059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45853 (local address 127.15.185.66:55595)
0927 04:01:09.573217 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:09.573267 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":61,"threads_started":1}
I20260927 04:01:09.700192 31410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:09.699945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45853 (local address 127.15.185.67:58207)
0927 04:01:09.700105 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:09.700148 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":80,"threads_started":1}
I20260927 04:01:10.490118 31411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:10.489848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45853 (local address 127.15.185.65:55881)
0927 04:01:10.490012 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:10.490066 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":69,"threads_started":1}
I20260927 04:01:10.574203 31412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:10.573983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45853 (local address 127.15.185.66:50239)
0927 04:01:10.574108 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:10.574154 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":62,"threads_started":1}
I20260927 04:01:10.701066 31413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:10.700845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45853 (local address 127.15.185.67:57377)
0927 04:01:10.700970 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:10.701017 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":61,"threads_started":1}
I20260927 04:01:11.490954 31414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:11.490686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45853 (local address 127.15.185.65:45883)
0927 04:01:11.490836 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:11.490887 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":64,"threads_started":1}
I20260927 04:01:11.575176 31415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:11.574917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45853 (local address 127.15.185.66:51281)
0927 04:01:11.575037 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:11.575126 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":57,"threads_started":1}
I20260927 04:01:11.702003 31416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:11.701693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45853 (local address 127.15.185.67:58773)
0927 04:01:11.701904 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:11.701955 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":124,"threads_started":1}
I20260927 04:01:12.491844 31417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:12.491602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45853 (local address 127.15.185.65:53447)
0927 04:01:12.491752 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:12.491792 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":63,"threads_started":1}
I20260927 04:01:12.576007 31418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:12.575777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45853 (local address 127.15.185.66:49673)
0927 04:01:12.575922 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:12.575961 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":71,"threads_started":1}
I20260927 04:01:12.698235 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 30940
I20260927 04:01:12.702907 31419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:12.702627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45853 (local address 127.15.185.67:42725)
0927 04:01:12.702800 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:12.702846 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:45853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":68,"threads_started":1}
I20260927 04:01:12.703220 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 31104
I20260927 04:01:12.707455 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 31236
2026-09-27T04:01:12Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18274 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:01:12 dist-test-slave-glgz krb5kdc[31422](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:01:12 dist-test-slave-glgz krb5kdc[31422](info): set up 2 sockets
Sep 27 04:01:12 dist-test-slave-glgz krb5kdc[31422](info): commencing operation
krb5kdc: starting...
W20260927 04:01:13.751591 16101 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Sep 27 04:01:13 dist-test-slave-glgz krb5kdc[31422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:01:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:01:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.185.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab.
Entry for principal kudu/127.15.185.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab.
I20260927 04:01:13.874533 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:33279
--webserver_interface=127.15.185.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:39459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.185.126:33279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:01:13.957508 31435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:01:13.957798 31435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:01:13.957840 31435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:01:13.959081 31435 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 04:01:13.959151 31435 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 04:01:13.959173 31435 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 04:01:13.959192 31435 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:01:13.959226 31435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:01:13.959252 31435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:01:13.959272 31435 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:01:13.959290 31435 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:01:13.960817 31435 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:39459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.15.185.126:33279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:33279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:01:13.961109 31435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:01:13.961350 31435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:01:13.966737 31441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:01:13.966792 31440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:01:13.966760 31435 server_base.cc:1034] running on GCE node
W20260927 04:01:13.966780 31443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:01:13.967401 31435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:01:13.967754 31435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:01:13.968871 31435 hybrid_clock.cc:648] HybridClock initialized: now 1790481673968857 us; error 32 us; skew 500 ppm
Sep 27 04:01:13 dist-test-slave-glgz krb5kdc[31422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481673, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:01:13.971561 31435 init.cc:377] Logged in from keytab as kudu/127.15.185.126@KRBTEST.COM (short username kudu)
I20260927 04:01:13.972095 31435 webserver.cc:460] Webserver started at http://127.15.185.126:45685/ using document root <none> and password file <none>
I20260927 04:01:13.972347 31435 fs_manager.cc:359] Metadata directory not provided
I20260927 04:01:13.972419 31435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:01:13.972577 31435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:01:13.973686 31435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/master-0/data/instance:
uuid: "6c975de4ec944e8e8fad12e086d292e9"
format_stamp: "Formatted at 2026-09-27 04:01:13 on dist-test-slave-glgz"
server_key: "8b1376547d4f7b1cf388fbd01c0dfe45"
server_key_iv: "e943c1a811e2b8bf79813ad9af0db01c"
server_key_version: "encryptionkey@0"
I20260927 04:01:13.974066 31435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/master-0/wal/instance:
uuid: "6c975de4ec944e8e8fad12e086d292e9"
format_stamp: "Formatted at 2026-09-27 04:01:13 on dist-test-slave-glgz"
server_key: "8b1376547d4f7b1cf388fbd01c0dfe45"
server_key_iv: "e943c1a811e2b8bf79813ad9af0db01c"
server_key_version: "encryptionkey@0"
I20260927 04:01:13.975492 31435 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 04:01:13.976225 31450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:01:13.976392 31435 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 04:01:13.976496 31435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/master-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/master-0/wal
uuid: "6c975de4ec944e8e8fad12e086d292e9"
format_stamp: "Formatted at 2026-09-27 04:01:13 on dist-test-slave-glgz"
server_key: "8b1376547d4f7b1cf388fbd01c0dfe45"
server_key_iv: "e943c1a811e2b8bf79813ad9af0db01c"
server_key_version: "encryptionkey@0"
I20260927 04:01:13.976570 31435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:01:14.004596 31435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:01:14.007146 31435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:01:14.007329 31435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:01:14.011263 31435 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.126:33279
I20260927 04:01:14.011312 31502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.126:33279 every 8 connection(s)
I20260927 04:01:14.011714 31435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
I20260927 04:01:14.012596 31503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:01:14.014986 31503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9: Bootstrap starting.
I20260927 04:01:14.015647 31503 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9: Neither blocks nor log segments found. Creating new log.
I20260927 04:01:14.015973 31503 log.cc:824] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9: Log is configured to *not* fsync() on all Append() calls
I20260927 04:01:14.016703 31503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9: No bootstrap required, opened a new log
I20260927 04:01:14.018313 31503 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c975de4ec944e8e8fad12e086d292e9" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 33279 } }
I20260927 04:01:14.018460 31503 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:01:14.018482 31503 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c975de4ec944e8e8fad12e086d292e9, State: Initialized, Role: FOLLOWER
I20260927 04:01:14.018667 31503 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c975de4ec944e8e8fad12e086d292e9" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 33279 } }
I20260927 04:01:14.018733 31503 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:01:14.018755 31503 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:01:14.018831 31503 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:01:14.019472 31503 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c975de4ec944e8e8fad12e086d292e9" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 33279 } }
I20260927 04:01:14.019601 31503 leader_election.cc:302] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c975de4ec944e8e8fad12e086d292e9; no voters:
I20260927 04:01:14.019785 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 31435
I20260927 04:01:14.019840 31503 leader_election.cc:288] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:01:14.019868 31506 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:01:14.019909 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/master-0/wal/instance
I20260927 04:01:14.020045 31506 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9 [term 1 LEADER]: Becoming Leader. State: Replica: 6c975de4ec944e8e8fad12e086d292e9, State: Running, Role: LEADER
I20260927 04:01:14.020120 31503 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:01:14.020143 16101 external_mini_cluster.cc:1428] Setting key a1395c7e57655136d9a2d1fa3627d46f
I20260927 04:01:14.020244 31506 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c975de4ec944e8e8fad12e086d292e9" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 33279 } }
I20260927 04:01:14.021353 31508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c975de4ec944e8e8fad12e086d292e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c975de4ec944e8e8fad12e086d292e9" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 33279 } } }
I20260927 04:01:14.021462 31508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9 [sys.catalog]: This master's current role is: LEADER
I20260927 04:01:14.021582 31508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c975de4ec944e8e8fad12e086d292e9. Latest consensus state: current_term: 1 leader_uuid: "6c975de4ec944e8e8fad12e086d292e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c975de4ec944e8e8fad12e086d292e9" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 33279 } } }
I20260927 04:01:14.021641 31508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9 [sys.catalog]: This master's current role is: LEADER
I20260927 04:01:14.021919 31522 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:01:14.022505 31522 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:01:14.024030 31522 catalog_manager.cc:1347] Generated new cluster ID: b1270b1b787d43b28a6652a59d51a2bc
I20260927 04:01:14.024084 31522 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 04:01:14 dist-test-slave-glgz krb5kdc[31422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 04:01:14.031071 31522 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:01:14.031522 31510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:14.020637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41202 (local address 127.15.185.126:33279)
0927 04:01:14.020858 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 04:01:14.020870 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 04:01:14.020908 (+ 38us) server_negotiation.cc:407] Connection header received
0927 04:01:14.021039 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:01:14.021048 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:01:14.021137 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:01:14.022152 (+ 1015us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:01:14.022278 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:01:14.023383 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:01:14.024425 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:01:14.024844 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:01:14.024975 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:01:14.027427 (+ 2452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:01:14.027432 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:01:14.027432 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:01:14.027444 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:01:14.029580 (+ 2136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:01:14.029974 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:01:14.029977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:01:14.029978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:01:14.030080 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:01:14.030294 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:01:14.030295 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:01:14.030296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:01:14.030578 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:01:14.031168 (+ 590us) server_negotiation.cc:1035] Waiting for connection context
0927 04:01:14.031235 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 04:01:14.031347 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":86,"threads_started":1}
I20260927 04:01:14.031844 31522 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:01:14.043514 31522 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9: Generated new TSK 0
I20260927 04:01:14.043751 31522 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.185.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab.
Entry for principal kudu/127.15.185.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab.
I20260927 04:01:14.073971 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.65:0
--local_ip_for_outbound_sockets=127.15.185.65
--webserver_interface=127.15.185.65
--webserver_port=0
--tserver_master_addrs=127.15.185.126:33279
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab
--principal=kudu/127.15.185.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:39459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:01:14.157095 31530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:01:14.157364 31530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:01:14.157407 31530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:01:14.158622 31530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:01:14.158702 31530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:01:14.158749 31530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.65
I20260927 04:01:14.160336 31530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:39459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab
--principal=kudu/127.15.185.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.185.65
--webserver_port=0
--tserver_master_addrs=127.15.185.126:33279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.65
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:01:14.160624 31530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:01:14.160871 31530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:01:14.166146 31538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:01:14.166213 31536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:01:14.166217 31535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:01:14.166334 31530 server_base.cc:1034] running on GCE node
I20260927 04:01:14.166620 31530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:01:14.166905 31530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:01:14.168058 31530 hybrid_clock.cc:648] HybridClock initialized: now 1790481674168045 us; error 31 us; skew 500 ppm
Sep 27 04:01:14 dist-test-slave-glgz krb5kdc[31422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481674, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:01:14.170686 31530 init.cc:377] Logged in from keytab as kudu/127.15.185.65@KRBTEST.COM (short username kudu)
I20260927 04:01:14.171219 31530 webserver.cc:460] Webserver started at http://127.15.185.65:40253/ using document root <none> and password file <none>
I20260927 04:01:14.171468 31530 fs_manager.cc:359] Metadata directory not provided
I20260927 04:01:14.171536 31530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:01:14.171672 31530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:01:14.172765 31530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-0/data/instance:
uuid: "df1bddad33464b0891abf90e06caf1b4"
format_stamp: "Formatted at 2026-09-27 04:01:14 on dist-test-slave-glgz"
server_key: "b102ecdbbbc3ad387c09d5fc28a1fbbb"
server_key_iv: "f848ed31dbfa4c943cc0616a80c68501"
server_key_version: "encryptionkey@0"
I20260927 04:01:14.173142 31530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-0/wal/instance:
uuid: "df1bddad33464b0891abf90e06caf1b4"
format_stamp: "Formatted at 2026-09-27 04:01:14 on dist-test-slave-glgz"
server_key: "b102ecdbbbc3ad387c09d5fc28a1fbbb"
server_key_iv: "f848ed31dbfa4c943cc0616a80c68501"
server_key_version: "encryptionkey@0"
I20260927 04:01:14.174525 31530 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 04:01:14.175310 31545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:01:14.175488 31530 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:01:14.175585 31530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-0/wal
uuid: "df1bddad33464b0891abf90e06caf1b4"
format_stamp: "Formatted at 2026-09-27 04:01:14 on dist-test-slave-glgz"
server_key: "b102ecdbbbc3ad387c09d5fc28a1fbbb"
server_key_iv: "f848ed31dbfa4c943cc0616a80c68501"
server_key_version: "encryptionkey@0"
I20260927 04:01:14.175668 31530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:01:14.223870 31530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:01:14.225966 31530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:01:14.226156 31530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:01:14.226416 31530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:01:14.226770 31530 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:01:14.226826 31530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:01:14.226867 31530 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:01:14.226909 31530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:01:14.233309 31530 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.65:39393
I20260927 04:01:14.233352 31658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.65:39393 every 8 connection(s)
I20260927 04:01:14.233728 31530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
I20260927 04:01:14.240110 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 31530
I20260927 04:01:14.240244 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-0/wal/instance
I20260927 04:01:14.240464 16101 external_mini_cluster.cc:1428] Setting key 9b28c6f191e987125623ffd6028bd191
Sep 27 04:01:14 dist-test-slave-glgz krb5kdc[31422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481674, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.65@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 04:01:14.244848 31510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:14.234503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:35303 (local address 127.15.185.126:33279)
0927 04:01:14.234593 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0927 04:01:14.234596 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:01:14.234783 (+ 187us) server_negotiation.cc:407] Connection header received
0927 04:01:14.235378 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:01:14.235379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:01:14.235390 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:01:14.235445 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:01:14.236390 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:01:14.236904 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:01:14.237913 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:01:14.238115 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:01:14.238173 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:01:14.241074 (+ 2901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:01:14.241075 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:01:14.241075 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:01:14.241083 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:01:14.242944 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:01:14.243643 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:01:14.243645 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:01:14.243645 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:01:14.243705 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:01:14.244050 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:01:14.244051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:01:14.244051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:01:14.244163 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:01:14.244210 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 04:01:14.244609 (+ 399us) server_negotiation.cc:299] Negotiation successful
0927 04:01:14.244708 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 04:01:14.244947 31661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:14.234452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:33279 (local address 127.15.185.65:35303)
0927 04:01:14.234634 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:14.234688 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 04:01:14.235278 (+ 590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:01:14.235505 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:01:14.235508 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:01:14.236096 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:01:14.236307 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:01:14.236312 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:01:14.236988 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:01:14.236989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:01:14.237823 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:01:14.237825 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:01:14.238207 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:01:14.238208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:01:14.238322 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:01:14.239021 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:01:14.239031 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:01:14.240926 (+ 1895us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:01:14.243100 (+ 2174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:01:14.243113 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:01:14.243117 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:01:14.243549 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:01:14.243788 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:01:14.243789 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:01:14.243790 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:01:14.243973 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:01:14.244232 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:01:14.244240 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:01:14.244448 (+ 208us) client_negotiation.cc:769] Sending connection context
0927 04:01:14.244543 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 04:01:14.244703 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":102,"threads_started":1}
I20260927 04:01:14.245579 31659 heartbeater.cc:347] Connected to a master server at 127.15.185.126:33279
I20260927 04:01:14.245720 31659 heartbeater.cc:464] Registering TS with master...
I20260927 04:01:14.245946 31659 heartbeater.cc:511] Master 127.15.185.126:33279 requested a full tablet report, sending...
I20260927 04:01:14.246524 31467 ts_manager.cc:194] Registered new tserver with Master: df1bddad33464b0891abf90e06caf1b4 (127.15.185.65:39393)
I20260927 04:01:14.247469 31467 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.185.65@KRBTEST.COM'} at 127.15.185.65:35303
WARNING: no policy specified for kudu/127.15.185.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab.
Entry for principal kudu/127.15.185.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab.
I20260927 04:01:14.269834 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.66:0
--local_ip_for_outbound_sockets=127.15.185.66
--webserver_interface=127.15.185.66
--webserver_port=0
--tserver_master_addrs=127.15.185.126:33279
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab
--principal=kudu/127.15.185.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:39459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:01:14.351912 31664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:01:14.352137 31664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:01:14.352164 31664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:01:14.353366 31664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:01:14.353411 31664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:01:14.353437 31664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.66
I20260927 04:01:14.354923 31664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:39459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab
--principal=kudu/127.15.185.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.185.66
--webserver_port=0
--tserver_master_addrs=127.15.185.126:33279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.66
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:01:14.355208 31664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:01:14.355466 31664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:01:14.360510 31672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:01:14.360610 31669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:01:14.360700 31664 server_base.cc:1034] running on GCE node
W20260927 04:01:14.360528 31670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:01:14.360944 31664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:01:14.361147 31664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:01:14.362322 31664 hybrid_clock.cc:648] HybridClock initialized: now 1790481674362315 us; error 35 us; skew 500 ppm
Sep 27 04:01:14 dist-test-slave-glgz krb5kdc[31422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481674, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:01:14.364835 31664 init.cc:377] Logged in from keytab as kudu/127.15.185.66@KRBTEST.COM (short username kudu)
I20260927 04:01:14.365269 31664 webserver.cc:460] Webserver started at http://127.15.185.66:42655/ using document root <none> and password file <none>
I20260927 04:01:14.365516 31664 fs_manager.cc:359] Metadata directory not provided
I20260927 04:01:14.365584 31664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:01:14.365734 31664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:01:14.366796 31664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-1/data/instance:
uuid: "10004b9896fe4d70a89becb761545a44"
format_stamp: "Formatted at 2026-09-27 04:01:14 on dist-test-slave-glgz"
server_key: "3fccf75ffae8b077638b18c173b4d921"
server_key_iv: "f6b8f0c8674b473502a9526e89d2b31d"
server_key_version: "encryptionkey@0"
I20260927 04:01:14.367189 31664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-1/wal/instance:
uuid: "10004b9896fe4d70a89becb761545a44"
format_stamp: "Formatted at 2026-09-27 04:01:14 on dist-test-slave-glgz"
server_key: "3fccf75ffae8b077638b18c173b4d921"
server_key_iv: "f6b8f0c8674b473502a9526e89d2b31d"
server_key_version: "encryptionkey@0"
I20260927 04:01:14.368673 31664 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:01:14.369398 31679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:01:14.369560 31664 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 04:01:14.369657 31664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-1/wal
uuid: "10004b9896fe4d70a89becb761545a44"
format_stamp: "Formatted at 2026-09-27 04:01:14 on dist-test-slave-glgz"
server_key: "3fccf75ffae8b077638b18c173b4d921"
server_key_iv: "f6b8f0c8674b473502a9526e89d2b31d"
server_key_version: "encryptionkey@0"
I20260927 04:01:14.369755 31664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:01:14.386845 31664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:01:14.389350 31664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:01:14.389523 31664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:01:14.389770 31664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:01:14.390097 31664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:01:14.390152 31664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:01:14.390187 31664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:01:14.390213 31664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:01:14.396271 31664 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.66:33351
I20260927 04:01:14.396328 31792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.66:33351 every 8 connection(s)
I20260927 04:01:14.396683 31664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
Sep 27 04:01:14 dist-test-slave-glgz krb5kdc[31422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481674, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.66@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 04:01:14.405119 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 31664
I20260927 04:01:14.405238 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-1/wal/instance
I20260927 04:01:14.405496 16101 external_mini_cluster.cc:1428] Setting key 15e6dd75d0c29a5d49a132eb599ef30b
I20260927 04:01:14.406498 31510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:14.397494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:57151 (local address 127.15.185.126:33279)
0927 04:01:14.397566 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0927 04:01:14.397569 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:01:14.397793 (+ 224us) server_negotiation.cc:407] Connection header received
0927 04:01:14.398300 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:01:14.398301 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:01:14.398311 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:01:14.398365 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:01:14.399315 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:01:14.399855 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:01:14.400847 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:01:14.401043 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:01:14.401120 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:01:14.403580 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:01:14.403582 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:01:14.403582 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:01:14.403589 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:01:14.404843 (+ 1254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:01:14.405370 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:01:14.405371 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:01:14.405371 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:01:14.405425 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:01:14.405704 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:01:14.405705 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:01:14.405705 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:01:14.405856 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:01:14.405945 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 04:01:14.406263 (+ 318us) server_negotiation.cc:299] Negotiation successful
0927 04:01:14.406374 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 04:01:14.406580 31795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:14.397468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:33279 (local address 127.15.185.66:57151)
0927 04:01:14.397667 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:14.397703 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 04:01:14.398198 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:01:14.398416 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:01:14.398419 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:01:14.398976 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:01:14.399197 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:01:14.399202 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:01:14.399942 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:01:14.399943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:01:14.400750 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:01:14.400752 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:01:14.401166 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:01:14.401167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:01:14.401291 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:01:14.401963 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:01:14.401973 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:01:14.403487 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:01:14.404942 (+ 1455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:01:14.404951 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:01:14.404953 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:01:14.405273 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:01:14.405491 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:01:14.405492 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:01:14.405492 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:01:14.405638 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:01:14.405938 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:01:14.405944 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:01:14.406088 (+ 144us) client_negotiation.cc:769] Sending connection context
0927 04:01:14.406224 (+ 136us) client_negotiation.cc:240] Negotiation successful
0927 04:01:14.406386 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":117,"threads_started":1}
I20260927 04:01:14.407013 31793 heartbeater.cc:347] Connected to a master server at 127.15.185.126:33279
I20260927 04:01:14.407148 31793 heartbeater.cc:464] Registering TS with master...
I20260927 04:01:14.407331 31793 heartbeater.cc:511] Master 127.15.185.126:33279 requested a full tablet report, sending...
I20260927 04:01:14.407795 31467 ts_manager.cc:194] Registered new tserver with Master: 10004b9896fe4d70a89becb761545a44 (127.15.185.66:33351)
I20260927 04:01:14.408181 31467 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.185.66@KRBTEST.COM'} at 127.15.185.66:57151
WARNING: no policy specified for kudu/127.15.185.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab.
Entry for principal kudu/127.15.185.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab.
I20260927 04:01:14.434857 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.67:0
--local_ip_for_outbound_sockets=127.15.185.67
--webserver_interface=127.15.185.67
--webserver_port=0
--tserver_master_addrs=127.15.185.126:33279
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab
--principal=kudu/127.15.185.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:39459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:01:14.525321 31798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:01:14.525588 31798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:01:14.525615 31798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:01:14.526960 31798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:01:14.527014 31798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:01:14.527114 31798 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.67
I20260927 04:01:14.528901 31798 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:39459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab
--principal=kudu/127.15.185.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.185.67
--webserver_port=0
--tserver_master_addrs=127.15.185.126:33279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31798
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.67
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:01:14.529186 31798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:01:14.529484 31798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:01:14.535040 31803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:01:14.535108 31804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:01:14.535322 31806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:01:14.535204 31798 server_base.cc:1034] running on GCE node
I20260927 04:01:14.535547 31798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:01:14.535796 31798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:01:14.536978 31798 hybrid_clock.cc:648] HybridClock initialized: now 1790481674536972 us; error 37 us; skew 500 ppm
Sep 27 04:01:14 dist-test-slave-glgz krb5kdc[31422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481674, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:01:14.539541 31798 init.cc:377] Logged in from keytab as kudu/127.15.185.67@KRBTEST.COM (short username kudu)
I20260927 04:01:14.540107 31798 webserver.cc:460] Webserver started at http://127.15.185.67:43525/ using document root <none> and password file <none>
I20260927 04:01:14.540302 31798 fs_manager.cc:359] Metadata directory not provided
I20260927 04:01:14.540343 31798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:01:14.540434 31798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:01:14.541481 31798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-2/data/instance:
uuid: "85dd61e73f4747529e1c38c05ae79d91"
format_stamp: "Formatted at 2026-09-27 04:01:14 on dist-test-slave-glgz"
server_key: "d9da12f34d9db010d0142328d019cdcf"
server_key_iv: "cc190ec569713feac05a30961e495d38"
server_key_version: "encryptionkey@0"
I20260927 04:01:14.541796 31798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-2/wal/instance:
uuid: "85dd61e73f4747529e1c38c05ae79d91"
format_stamp: "Formatted at 2026-09-27 04:01:14 on dist-test-slave-glgz"
server_key: "d9da12f34d9db010d0142328d019cdcf"
server_key_iv: "cc190ec569713feac05a30961e495d38"
server_key_version: "encryptionkey@0"
I20260927 04:01:14.543183 31798 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 04:01:14.544123 31813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:01:14.544349 31798 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:01:14.544448 31798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-2/wal
uuid: "85dd61e73f4747529e1c38c05ae79d91"
format_stamp: "Formatted at 2026-09-27 04:01:14 on dist-test-slave-glgz"
server_key: "d9da12f34d9db010d0142328d019cdcf"
server_key_iv: "cc190ec569713feac05a30961e495d38"
server_key_version: "encryptionkey@0"
I20260927 04:01:14.544524 31798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:01:14.563555 31798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:01:14.565536 31798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:01:14.565692 31798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:01:14.565945 31798 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:01:14.566270 31798 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:01:14.566303 31798 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:01:14.566349 31798 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:01:14.566376 31798 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:01:14.572546 31798 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.67:39469
I20260927 04:01:14.572588 31926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.67:39469 every 8 connection(s)
I20260927 04:01:14.572939 31798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
Sep 27 04:01:14 dist-test-slave-glgz krb5kdc[31422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481674, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.67@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 04:01:14.580271 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 31798
I20260927 04:01:14.580382 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-2/wal/instance
I20260927 04:01:14.580586 16101 external_mini_cluster.cc:1428] Setting key f3f038d967b79a3afa3e0902fa33e7e5
I20260927 04:01:14.583074 31510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:14.573712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:45503 (local address 127.15.185.126:33279)
0927 04:01:14.573805 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0927 04:01:14.573808 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:01:14.574005 (+ 197us) server_negotiation.cc:407] Connection header received
0927 04:01:14.574546 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:01:14.574548 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:01:14.574560 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:01:14.574617 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:01:14.575626 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:01:14.576145 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:01:14.577145 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:01:14.577360 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:01:14.577437 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:01:14.579720 (+ 2283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:01:14.579722 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:01:14.579722 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:01:14.579729 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:01:14.581370 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:01:14.581888 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:01:14.581889 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:01:14.581890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:01:14.581971 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:01:14.582290 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:01:14.582291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:01:14.582291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:01:14.582430 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:01:14.582517 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 04:01:14.582851 (+ 334us) server_negotiation.cc:299] Negotiation successful
0927 04:01:14.582944 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 04:01:14.583320 31929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:14.573704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:33279 (local address 127.15.185.67:45503)
0927 04:01:14.573884 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:14.573922 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 04:01:14.574436 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:01:14.574700 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:01:14.574703 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:01:14.575325 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:01:14.575521 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:01:14.575526 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:01:14.576244 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:01:14.576245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:01:14.577036 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:01:14.577038 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:01:14.577455 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:01:14.577456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:01:14.577580 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:01:14.578098 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:01:14.578106 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:01:14.579616 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:01:14.581429 (+ 1813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:01:14.581440 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:01:14.581443 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:01:14.581714 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:01:14.582053 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:01:14.582054 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:01:14.582055 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:01:14.582207 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:01:14.582506 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:01:14.582523 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:01:14.582679 (+ 156us) client_negotiation.cc:769] Sending connection context
0927 04:01:14.582826 (+ 147us) client_negotiation.cc:240] Negotiation successful
0927 04:01:14.583007 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":119,"threads_started":1}
I20260927 04:01:14.583899 31927 heartbeater.cc:347] Connected to a master server at 127.15.185.126:33279
I20260927 04:01:14.583987 31927 heartbeater.cc:464] Registering TS with master...
I20260927 04:01:14.584141 31927 heartbeater.cc:511] Master 127.15.185.126:33279 requested a full tablet report, sending...
I20260927 04:01:14.584514 31467 ts_manager.cc:194] Registered new tserver with Master: 85dd61e73f4747529e1c38c05ae79d91 (127.15.185.67:39469)
I20260927 04:01:14.584926 31467 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.185.67@KRBTEST.COM'} at 127.15.185.67:45503
I20260927 04:01:14.592464 16101 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 04:01:14.600211 31510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:14.593961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41218 (local address 127.15.185.126:33279)
0927 04:01:14.594054 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0927 04:01:14.594056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:01:14.594164 (+ 108us) server_negotiation.cc:407] Connection header received
0927 04:01:14.594233 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:01:14.594234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:01:14.594245 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:01:14.594311 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:01:14.594913 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:01:14.595491 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:01:14.596231 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:01:14.596449 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:01:14.596527 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:01:14.597697 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:01:14.597698 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:01:14.597699 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:01:14.597707 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:01:14.598943 (+ 1236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:01:14.599346 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:01:14.599348 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:01:14.599348 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:01:14.599409 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:01:14.599618 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:01:14.599619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:01:14.599619 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:01:14.599781 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:01:14.599887 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 04:01:14.600020 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 04:01:14.600105 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 04:01:14.601770 31467 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41218:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 04:01:14.603402 31467 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 04:01:14.610248 31938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:14.605955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.67:39469 (local address 127.0.0.1:54856)
0927 04:01:14.606588 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:14.606598 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 04:01:14.606651 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:01:14.606864 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:01:14.606864 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:01:14.606866 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:01:14.606903 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:01:14.606904 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:01:14.607804 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:01:14.607805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:01:14.609164 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:01:14.609166 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:01:14.610028 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:01:14.610029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:01:14.610080 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:01:14.610084 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 04:01:14.610136 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 04:01:14.610177 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"mutex_wait_us":53,"thread_start_us":69,"threads_started":1}
I20260927 04:01:14.610280 31942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:14.606065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54856 (local address 127.15.185.67:39469)
0927 04:01:14.606271 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 04:01:14.606276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:01:14.606652 (+ 376us) server_negotiation.cc:407] Connection header received
0927 04:01:14.606720 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:01:14.606724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:01:14.606739 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:01:14.606821 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:01:14.606953 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:01:14.607697 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:01:14.609261 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:01:14.609958 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:01:14.610028 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:01:14.610079 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 04:01:14.610151 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 04:01:14.610192 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":100,"threads_started":1}
I20260927 04:01:14.610380 31940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:14.605851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38034 (local address 127.15.185.65:39393)
0927 04:01:14.606054 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 04:01:14.606059 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:01:14.606223 (+ 164us) server_negotiation.cc:407] Connection header received
0927 04:01:14.606276 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:01:14.606279 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:01:14.606296 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:01:14.606434 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:01:14.606726 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:01:14.607887 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:01:14.609268 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:01:14.609755 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:01:14.609857 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:01:14.609923 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 04:01:14.610224 (+ 301us) server_negotiation.cc:299] Negotiation successful
0927 04:01:14.610274 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":80,"threads_started":1}
I20260927 04:01:14.610414 31943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:14.606106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37364 (local address 127.15.185.66:33351)
0927 04:01:14.606328 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 04:01:14.606332 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:01:14.606341 (+ 9us) server_negotiation.cc:407] Connection header received
0927 04:01:14.606382 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:01:14.606384 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:01:14.606419 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:01:14.606494 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:01:14.606844 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:01:14.607942 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:01:14.609085 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:01:14.609832 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:01:14.609905 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:01:14.609964 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 04:01:14.610291 (+ 327us) server_negotiation.cc:299] Negotiation successful
0927 04:01:14.610343 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":112,"threads_started":1}
I20260927 04:01:14.610181 31941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:14.605727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.65:39393 (local address 127.0.0.1:38034)
0927 04:01:14.606108 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:14.606119 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 04:01:14.606186 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:01:14.606541 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:01:14.606544 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:01:14.606547 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:01:14.606638 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:01:14.606640 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:01:14.607989 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:01:14.607990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:01:14.609164 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:01:14.609166 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:01:14.609860 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:01:14.609861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:01:14.609953 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:01:14.609966 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 04:01:14.610036 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 04:01:14.610091 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":7,"thread_start_us":95,"threads_started":1}
I20260927 04:01:14.610626 31939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:14.605690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.66:33351 (local address 127.0.0.1:37364)
0927 04:01:14.605875 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:14.605918 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 04:01:14.605982 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:01:14.606731 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:01:14.606732 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:01:14.606734 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:01:14.606784 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:01:14.606786 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:01:14.608044 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:01:14.608045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:01:14.608996 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:01:14.608998 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:01:14.610001 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:01:14.610002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:01:14.610065 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:01:14.610069 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 04:01:14.610117 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 04:01:14.610174 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":61,"threads_started":1}
I20260927 04:01:14.611515 31593 tablet_service.cc:1467] Processing CreateTablet for tablet 1f20566ff1164b67bbd8c5bd70b4c9f8 (DEFAULT_TABLE table=test-table [id=786a1a7ffe024b07b1485d9431198551]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:01:14.611515 31861 tablet_service.cc:1467] Processing CreateTablet for tablet 1f20566ff1164b67bbd8c5bd70b4c9f8 (DEFAULT_TABLE table=test-table [id=786a1a7ffe024b07b1485d9431198551]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:01:14.611567 31727 tablet_service.cc:1467] Processing CreateTablet for tablet 1f20566ff1164b67bbd8c5bd70b4c9f8 (DEFAULT_TABLE table=test-table [id=786a1a7ffe024b07b1485d9431198551]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:01:14.611861 31727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f20566ff1164b67bbd8c5bd70b4c9f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:01:14.611877 31593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f20566ff1164b67bbd8c5bd70b4c9f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:01:14.611877 31861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f20566ff1164b67bbd8c5bd70b4c9f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:01:14.614804 31944 tablet_bootstrap.cc:492] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P 10004b9896fe4d70a89becb761545a44: Bootstrap starting.
I20260927 04:01:14.615463 31946 tablet_bootstrap.cc:492] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P df1bddad33464b0891abf90e06caf1b4: Bootstrap starting.
I20260927 04:01:14.615839 31944 tablet_bootstrap.cc:654] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P 10004b9896fe4d70a89becb761545a44: Neither blocks nor log segments found. Creating new log.
I20260927 04:01:14.616158 31946 tablet_bootstrap.cc:654] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P df1bddad33464b0891abf90e06caf1b4: Neither blocks nor log segments found. Creating new log.
I20260927 04:01:14.616503 31944 log.cc:824] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P 10004b9896fe4d70a89becb761545a44: Log is configured to *not* fsync() on all Append() calls
I20260927 04:01:14.616770 31946 log.cc:824] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P df1bddad33464b0891abf90e06caf1b4: Log is configured to *not* fsync() on all Append() calls
I20260927 04:01:14.616952 31945 tablet_bootstrap.cc:492] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P 85dd61e73f4747529e1c38c05ae79d91: Bootstrap starting.
I20260927 04:01:14.617456 31944 tablet_bootstrap.cc:492] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P 10004b9896fe4d70a89becb761545a44: No bootstrap required, opened a new log
I20260927 04:01:14.617545 31944 ts_tablet_manager.cc:1397] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P 10004b9896fe4d70a89becb761545a44: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 04:01:14.617563 31946 tablet_bootstrap.cc:492] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P df1bddad33464b0891abf90e06caf1b4: No bootstrap required, opened a new log
I20260927 04:01:14.617633 31946 ts_tablet_manager.cc:1397] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P df1bddad33464b0891abf90e06caf1b4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 04:01:14.617856 31945 tablet_bootstrap.cc:654] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P 85dd61e73f4747529e1c38c05ae79d91: Neither blocks nor log segments found. Creating new log.
I20260927 04:01:14.618398 31945 log.cc:824] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P 85dd61e73f4747529e1c38c05ae79d91: Log is configured to *not* fsync() on all Append() calls
I20260927 04:01:14.619244 31945 tablet_bootstrap.cc:492] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P 85dd61e73f4747529e1c38c05ae79d91: No bootstrap required, opened a new log
I20260927 04:01:14.619318 31945 ts_tablet_manager.cc:1397] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P 85dd61e73f4747529e1c38c05ae79d91: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 04:01:14.619498 31946 raft_consensus.cc:348] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P df1bddad33464b0891abf90e06caf1b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10004b9896fe4d70a89becb761545a44" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 33351 } } peers { permanent_uuid: "df1bddad33464b0891abf90e06caf1b4" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 39393 } } peers { permanent_uuid: "85dd61e73f4747529e1c38c05ae79d91" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 39469 } }
I20260927 04:01:14.619669 31946 raft_consensus.cc:374] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P df1bddad33464b0891abf90e06caf1b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:01:14.619730 31946 raft_consensus.cc:729] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P df1bddad33464b0891abf90e06caf1b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1bddad33464b0891abf90e06caf1b4, State: Initialized, Role: FOLLOWER
I20260927 04:01:14.619786 31944 raft_consensus.cc:348] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P 10004b9896fe4d70a89becb761545a44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10004b9896fe4d70a89becb761545a44" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 33351 } } peers { permanent_uuid: "df1bddad33464b0891abf90e06caf1b4" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 39393 } } peers { permanent_uuid: "85dd61e73f4747529e1c38c05ae79d91" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 39469 } }
I20260927 04:01:14.619966 31944 raft_consensus.cc:374] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P 10004b9896fe4d70a89becb761545a44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:01:14.619912 31946 consensus_queue.cc:260] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P df1bddad33464b0891abf90e06caf1b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10004b9896fe4d70a89becb761545a44" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 33351 } } peers { permanent_uuid: "df1bddad33464b0891abf90e06caf1b4" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 39393 } } peers { permanent_uuid: "85dd61e73f4747529e1c38c05ae79d91" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 39469 } }
I20260927 04:01:14.620041 31944 raft_consensus.cc:729] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P 10004b9896fe4d70a89becb761545a44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10004b9896fe4d70a89becb761545a44, State: Initialized, Role: FOLLOWER
I20260927 04:01:14.620200 31944 consensus_queue.cc:260] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P 10004b9896fe4d70a89becb761545a44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10004b9896fe4d70a89becb761545a44" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 33351 } } peers { permanent_uuid: "df1bddad33464b0891abf90e06caf1b4" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 39393 } } peers { permanent_uuid: "85dd61e73f4747529e1c38c05ae79d91" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 39469 } }
I20260927 04:01:14.620275 31946 ts_tablet_manager.cc:1428] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P df1bddad33464b0891abf90e06caf1b4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 04:01:14.620426 31659 heartbeater.cc:503] Master 127.15.185.126:33279 was elected leader, sending a full tablet report...
I20260927 04:01:14.620608 31944 ts_tablet_manager.cc:1428] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P 10004b9896fe4d70a89becb761545a44: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 04:01:14.620658 31793 heartbeater.cc:503] Master 127.15.185.126:33279 was elected leader, sending a full tablet report...
I20260927 04:01:14.621785 31945 raft_consensus.cc:348] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P 85dd61e73f4747529e1c38c05ae79d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10004b9896fe4d70a89becb761545a44" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 33351 } } peers { permanent_uuid: "df1bddad33464b0891abf90e06caf1b4" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 39393 } } peers { permanent_uuid: "85dd61e73f4747529e1c38c05ae79d91" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 39469 } }
I20260927 04:01:14.621937 31945 raft_consensus.cc:374] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P 85dd61e73f4747529e1c38c05ae79d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:01:14.621959 31945 raft_consensus.cc:729] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P 85dd61e73f4747529e1c38c05ae79d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85dd61e73f4747529e1c38c05ae79d91, State: Initialized, Role: FOLLOWER
I20260927 04:01:14.622085 31945 consensus_queue.cc:260] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P 85dd61e73f4747529e1c38c05ae79d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10004b9896fe4d70a89becb761545a44" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 33351 } } peers { permanent_uuid: "df1bddad33464b0891abf90e06caf1b4" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 39393 } } peers { permanent_uuid: "85dd61e73f4747529e1c38c05ae79d91" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 39469 } }
I20260927 04:01:14.622355 31945 ts_tablet_manager.cc:1428] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P 85dd61e73f4747529e1c38c05ae79d91: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 04:01:14.622419 31927 heartbeater.cc:503] Master 127.15.185.126:33279 was elected leader, sending a full tablet report...
W20260927 04:01:14.647294 31794 tablet.cc:2369] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P 10004b9896fe4d70a89becb761545a44: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 04:01:14.662330 31950 raft_consensus.cc:482] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P df1bddad33464b0891abf90e06caf1b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:01:14.662473 31950 raft_consensus.cc:504] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P df1bddad33464b0891abf90e06caf1b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10004b9896fe4d70a89becb761545a44" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 33351 } } peers { permanent_uuid: "df1bddad33464b0891abf90e06caf1b4" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 39393 } } peers { permanent_uuid: "85dd61e73f4747529e1c38c05ae79d91" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 39469 } }
I20260927 04:01:14.662889 31950 leader_election.cc:288] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P df1bddad33464b0891abf90e06caf1b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10004b9896fe4d70a89becb761545a44 (127.15.185.66:33351), 85dd61e73f4747529e1c38c05ae79d91 (127.15.185.67:39469)
I20260927 04:01:14.666216 31942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:14.663205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:44967 (local address 127.15.185.67:39469)
0927 04:01:14.663249 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 04:01:14.663251 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:01:14.663424 (+ 173us) server_negotiation.cc:407] Connection header received
0927 04:01:14.663454 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:01:14.663455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:01:14.663464 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:01:14.663528 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:01:14.663726 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:01:14.664463 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:01:14.665455 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:01:14.665844 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:01:14.665931 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:01:14.665941 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 04:01:14.666094 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 04:01:14.666136 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 04:01:14.666226 31661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:14.663118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.66:33351 (local address 127.15.185.65:34127)
0927 04:01:14.663174 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:14.663186 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 04:01:14.663237 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:01:14.663435 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:01:14.663437 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:01:14.663439 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:01:14.663523 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:01:14.663526 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:01:14.664551 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:01:14.664551 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:01:14.665353 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:01:14.665356 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:01:14.665935 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:01:14.665936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:01:14.666030 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:01:14.666033 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 04:01:14.666109 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 04:01:14.666149 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":32}
I20260927 04:01:14.666232 31943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:14.663156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:34127 (local address 127.15.185.66:33351)
0927 04:01:14.663205 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 04:01:14.663207 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:01:14.663224 (+ 17us) server_negotiation.cc:407] Connection header received
0927 04:01:14.663302 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:01:14.663303 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:01:14.663313 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:01:14.663406 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:01:14.663699 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:01:14.664470 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:01:14.665456 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:01:14.665847 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:01:14.665937 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:01:14.665946 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 04:01:14.666132 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 04:01:14.666171 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 04:01:14.666234 31953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:14.663150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.67:39469 (local address 127.15.185.65:44967)
0927 04:01:14.663334 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:14.663346 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 04:01:14.663406 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:01:14.663539 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:01:14.663540 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:01:14.663541 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:01:14.663629 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:01:14.663631 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:01:14.664568 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:01:14.664568 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:01:14.665357 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:01:14.665359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:01:14.665939 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:01:14.665940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:01:14.666019 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:01:14.666023 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 04:01:14.666063 (+ 40us) client_negotiation.cc:240] Negotiation successful
0927 04:01:14.666132 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":19,"thread_start_us":129,"threads_started":1}
I20260927 04:01:14.666569 31747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f20566ff1164b67bbd8c5bd70b4c9f8" candidate_uuid: "df1bddad33464b0891abf90e06caf1b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10004b9896fe4d70a89becb761545a44" is_pre_election: true
I20260927 04:01:14.666647 31881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f20566ff1164b67bbd8c5bd70b4c9f8" candidate_uuid: "df1bddad33464b0891abf90e06caf1b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85dd61e73f4747529e1c38c05ae79d91" is_pre_election: true
I20260927 04:01:14.666760 31747 raft_consensus.cc:2454] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P 10004b9896fe4d70a89becb761545a44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1bddad33464b0891abf90e06caf1b4 in term 0.
I20260927 04:01:14.666765 31881 raft_consensus.cc:2454] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P 85dd61e73f4747529e1c38c05ae79d91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1bddad33464b0891abf90e06caf1b4 in term 0.
I20260927 04:01:14.666987 31549 leader_election.cc:302] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P df1bddad33464b0891abf90e06caf1b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85dd61e73f4747529e1c38c05ae79d91, df1bddad33464b0891abf90e06caf1b4; no voters:
I20260927 04:01:14.667160 31950 raft_consensus.cc:2781] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P df1bddad33464b0891abf90e06caf1b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 04:01:14.667248 31950 raft_consensus.cc:482] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P df1bddad33464b0891abf90e06caf1b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 04:01:14.667285 31950 raft_consensus.cc:3037] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P df1bddad33464b0891abf90e06caf1b4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:01:14.668056 31950 raft_consensus.cc:504] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P df1bddad33464b0891abf90e06caf1b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10004b9896fe4d70a89becb761545a44" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 33351 } } peers { permanent_uuid: "df1bddad33464b0891abf90e06caf1b4" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 39393 } } peers { permanent_uuid: "85dd61e73f4747529e1c38c05ae79d91" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 39469 } }
I20260927 04:01:14.668238 31950 leader_election.cc:288] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P df1bddad33464b0891abf90e06caf1b4 [CANDIDATE]: Term 1 election: Requested vote from peers 10004b9896fe4d70a89becb761545a44 (127.15.185.66:33351), 85dd61e73f4747529e1c38c05ae79d91 (127.15.185.67:39469)
I20260927 04:01:14.668417 31881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f20566ff1164b67bbd8c5bd70b4c9f8" candidate_uuid: "df1bddad33464b0891abf90e06caf1b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85dd61e73f4747529e1c38c05ae79d91"
I20260927 04:01:14.668511 31881 raft_consensus.cc:3037] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P 85dd61e73f4747529e1c38c05ae79d91 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:01:14.669319 31881 raft_consensus.cc:2454] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P 85dd61e73f4747529e1c38c05ae79d91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1bddad33464b0891abf90e06caf1b4 in term 1.
I20260927 04:01:14.669449 31747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f20566ff1164b67bbd8c5bd70b4c9f8" candidate_uuid: "df1bddad33464b0891abf90e06caf1b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10004b9896fe4d70a89becb761545a44"
I20260927 04:01:14.669482 31549 leader_election.cc:302] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P df1bddad33464b0891abf90e06caf1b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85dd61e73f4747529e1c38c05ae79d91, df1bddad33464b0891abf90e06caf1b4; no voters:
I20260927 04:01:14.669556 31747 raft_consensus.cc:3037] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P 10004b9896fe4d70a89becb761545a44 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:01:14.669600 31950 raft_consensus.cc:2781] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P df1bddad33464b0891abf90e06caf1b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:01:14.669811 31950 raft_consensus.cc:686] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P df1bddad33464b0891abf90e06caf1b4 [term 1 LEADER]: Becoming Leader. State: Replica: df1bddad33464b0891abf90e06caf1b4, State: Running, Role: LEADER
I20260927 04:01:14.669927 31950 consensus_queue.cc:237] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P df1bddad33464b0891abf90e06caf1b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10004b9896fe4d70a89becb761545a44" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 33351 } } peers { permanent_uuid: "df1bddad33464b0891abf90e06caf1b4" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 39393 } } peers { permanent_uuid: "85dd61e73f4747529e1c38c05ae79d91" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 39469 } }
I20260927 04:01:14.670455 31747 raft_consensus.cc:2454] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P 10004b9896fe4d70a89becb761545a44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1bddad33464b0891abf90e06caf1b4 in term 1.
I20260927 04:01:14.670869 31467 catalog_manager.cc:5495] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P df1bddad33464b0891abf90e06caf1b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to df1bddad33464b0891abf90e06caf1b4 (127.15.185.65). New cstate: current_term: 1 leader_uuid: "df1bddad33464b0891abf90e06caf1b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10004b9896fe4d70a89becb761545a44" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 33351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df1bddad33464b0891abf90e06caf1b4" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 39393 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85dd61e73f4747529e1c38c05ae79d91" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 39469 } health_report { overall_health: UNKNOWN } } }
I20260927 04:01:14.683530 31940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:14.680997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38036 (local address 127.15.185.65:39393)
0927 04:01:14.681058 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 04:01:14.681060 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:01:14.681070 (+ 10us) server_negotiation.cc:407] Connection header received
0927 04:01:14.681148 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:01:14.681149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:01:14.681169 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:01:14.681258 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:01:14.681404 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:01:14.681896 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:01:14.682698 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:01:14.682914 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:01:14.682991 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:01:14.683197 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:01:14.683341 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:01:14.683414 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 04:01:14.683452 (+ 38us) server_negotiation.cc:299] Negotiation successful
0927 04:01:14.683471 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 04:01:14.685433 31747 raft_consensus.cc:1261] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P 10004b9896fe4d70a89becb761545a44 [term 1 FOLLOWER]: Refusing update from remote peer df1bddad33464b0891abf90e06caf1b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:01:14.685433 31881 raft_consensus.cc:1261] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P 85dd61e73f4747529e1c38c05ae79d91 [term 1 FOLLOWER]: Refusing update from remote peer df1bddad33464b0891abf90e06caf1b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:01:14.685873 31950 consensus_queue.cc:1035] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P df1bddad33464b0891abf90e06caf1b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85dd61e73f4747529e1c38c05ae79d91" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 39469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:01:14.686035 31954 consensus_queue.cc:1035] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P df1bddad33464b0891abf90e06caf1b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10004b9896fe4d70a89becb761545a44" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 33351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:01:14.690577 31467 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:01:14.690694 31467 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:01:14.691454 31467 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9: Sending DeleteTablet for 3 replicas of tablet 1f20566ff1164b67bbd8c5bd70b4c9f8
I20260927 04:01:14.691774 31593 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f20566ff1164b67bbd8c5bd70b4c9f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:01:14 UTC) from {username='kudu', principal='kudu/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:38034
I20260927 04:01:14.691797 31727 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f20566ff1164b67bbd8c5bd70b4c9f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:01:14 UTC) from {username='kudu', principal='kudu/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:37364
I20260927 04:01:14.691839 31861 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f20566ff1164b67bbd8c5bd70b4c9f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:01:14 UTC) from {username='kudu', principal='kudu/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:54856
I20260927 04:01:14.692028 31967 tablet_replica.cc:330] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P df1bddad33464b0891abf90e06caf1b4: stopping tablet replica
I20260927 04:01:14.692067 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 31530
I20260927 04:01:14.692128 31967 raft_consensus.cc:2229] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P df1bddad33464b0891abf90e06caf1b4 [term 1 LEADER]: Raft consensus shutting down.
I20260927 04:01:14.692256 31969 tablet_replica.cc:330] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P 85dd61e73f4747529e1c38c05ae79d91: stopping tablet replica
I20260927 04:01:14.692384 31969 raft_consensus.cc:2229] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P 85dd61e73f4747529e1c38c05ae79d91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:01:14.692512 31969 raft_consensus.cc:2258] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P 85dd61e73f4747529e1c38c05ae79d91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:01:14.693089 31969 ts_tablet_manager.cc:1905] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P 85dd61e73f4747529e1c38c05ae79d91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:01:14.695185 31969 ts_tablet_manager.cc:1918] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P 85dd61e73f4747529e1c38c05ae79d91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:01:14.695261 31969 log.cc:1196] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P 85dd61e73f4747529e1c38c05ae79d91: Deleting WAL directory at /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-2/wal/wals/1f20566ff1164b67bbd8c5bd70b4c9f8
I20260927 04:01:14.695529 31969 ts_tablet_manager.cc:1939] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P 85dd61e73f4747529e1c38c05ae79d91: Deleting consensus metadata
I20260927 04:01:14.695983 31454 catalog_manager.cc:4841] TS 85dd61e73f4747529e1c38c05ae79d91 (127.15.185.67:39469): tablet 1f20566ff1164b67bbd8c5bd70b4c9f8 (table test-table [id=786a1a7ffe024b07b1485d9431198551]) successfully deleted
I20260927 04:01:14.698963 31968 tablet_replica.cc:330] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P 10004b9896fe4d70a89becb761545a44: stopping tablet replica
W20260927 04:01:14.698962 31454 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.185.65:39393 (error 108)
I20260927 04:01:14.699122 31968 raft_consensus.cc:2229] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P 10004b9896fe4d70a89becb761545a44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:01:14.699255 31968 raft_consensus.cc:2258] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P 10004b9896fe4d70a89becb761545a44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:01:14.699270 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 31664
I20260927 04:01:14.700057 31939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:14.699899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.65:39393 (local address 127.0.0.1:38044)
0927 04:01:14.699965 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:14.700012 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.65:39393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 04:01:14.700227 31454 catalog_manager.cc:4584] TS df1bddad33464b0891abf90e06caf1b4 (127.15.185.65:39393): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1f20566ff1164b67bbd8c5bd70b4c9f8: Network error: Client connection negotiation failed: client connection to 127.15.185.65:39393: connect: Connection refused (error 111)
I20260927 04:01:14.703975 31939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:14.703850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.66:33351 (local address 127.0.0.1:37380)
0927 04:01:14.703901 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:14.703935 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.66:33351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 04:01:14.704003 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 31798
I20260927 04:01:14.708555 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 31435
I20260927 04:01:34.713953 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:33279
--webserver_interface=127.15.185.126
--webserver_port=45685
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:39459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.185.126:33279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:01:34.798300 31971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:01:34.798550 31971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:01:34.798586 31971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:01:34.799830 31971 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 04:01:34.799899 31971 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 04:01:34.799922 31971 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 04:01:34.799942 31971 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:01:34.799974 31971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:01:34.800001 31971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:01:34.800027 31971 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:01:34.800056 31971 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:01:34.801535 31971 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:39459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.15.185.126:33279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:33279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.185.126
--webserver_port=45685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:01:34.801831 31971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:01:34.802064 31971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:01:34.811128 31978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:01:34.811187 31977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:01:34.811240 31971 server_base.cc:1034] running on GCE node
W20260927 04:01:34.811362 31980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:01:34.811707 31971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:01:34.812067 31971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:01:34.813251 31971 hybrid_clock.cc:648] HybridClock initialized: now 1790481694813236 us; error 50 us; skew 500 ppm
Sep 27 04:01:34 dist-test-slave-glgz krb5kdc[31422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481694, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:01:34.815850 31971 init.cc:377] Logged in from keytab as kudu/127.15.185.126@KRBTEST.COM (short username kudu)
I20260927 04:01:34.816262 31971 webserver.cc:460] Webserver started at http://127.15.185.126:45685/ using document root <none> and password file <none>
I20260927 04:01:34.816464 31971 fs_manager.cc:359] Metadata directory not provided
I20260927 04:01:34.816519 31971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:01:34.817865 31971 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:01:34.818547 31987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:01:34.818766 31971 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:01:34.818881 31971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/master-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/master-0/wal
uuid: "6c975de4ec944e8e8fad12e086d292e9"
format_stamp: "Formatted at 2026-09-27 04:01:13 on dist-test-slave-glgz"
server_key: "8b1376547d4f7b1cf388fbd01c0dfe45"
server_key_iv: "e943c1a811e2b8bf79813ad9af0db01c"
server_key_version: "encryptionkey@0"
I20260927 04:01:34.819380 31971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:01:34.842831 31971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:01:34.844928 31971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:01:34.845103 31971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:01:34.849316 31971 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.126:33279
I20260927 04:01:34.849356 32039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.126:33279 every 8 connection(s)
I20260927 04:01:34.849741 31971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
I20260927 04:01:34.851217 32040 sys_catalog.cc:263] Verifying existing consensus state
I20260927 04:01:34.851925 32040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9: Bootstrap starting.
I20260927 04:01:34.859447 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 31971
I20260927 04:01:34.859802 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.65:39393
--local_ip_for_outbound_sockets=127.15.185.65
--tserver_master_addrs=127.15.185.126:33279
--webserver_port=40253
--webserver_interface=127.15.185.65
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab
--principal=kudu/127.15.185.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:39459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 04:01:34.860406 32040 log.cc:824] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9: Log is configured to *not* fsync() on all Append() calls
I20260927 04:01:34.862785 32040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 04:01:34.863132 32040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9: Bootstrap complete.
I20260927 04:01:34.867208 32040 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c975de4ec944e8e8fad12e086d292e9" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 33279 } }
I20260927 04:01:34.867602 32040 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c975de4ec944e8e8fad12e086d292e9, State: Initialized, Role: FOLLOWER
I20260927 04:01:34.867807 32040 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c975de4ec944e8e8fad12e086d292e9" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 33279 } }
I20260927 04:01:34.867897 32040 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:01:34.867959 32040 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:01:34.868021 32040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9 [term 1 FOLLOWER]: Advancing to term 2
I20260927 04:01:34.869625 32040 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c975de4ec944e8e8fad12e086d292e9" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 33279 } }
I20260927 04:01:34.869760 32040 leader_election.cc:302] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c975de4ec944e8e8fad12e086d292e9; no voters:
I20260927 04:01:34.869940 32040 leader_election.cc:288] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 04:01:34.870015 32044 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 04:01:34.870172 32044 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9 [term 2 LEADER]: Becoming Leader. State: Replica: 6c975de4ec944e8e8fad12e086d292e9, State: Running, Role: LEADER
I20260927 04:01:34.870237 32040 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:01:34.870304 32044 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c975de4ec944e8e8fad12e086d292e9" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 33279 } }
I20260927 04:01:34.870743 32046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c975de4ec944e8e8fad12e086d292e9. Latest consensus state: current_term: 2 leader_uuid: "6c975de4ec944e8e8fad12e086d292e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c975de4ec944e8e8fad12e086d292e9" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 33279 } } }
I20260927 04:01:34.870741 32045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6c975de4ec944e8e8fad12e086d292e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c975de4ec944e8e8fad12e086d292e9" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 33279 } } }
I20260927 04:01:34.870855 32046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9 [sys.catalog]: This master's current role is: LEADER
I20260927 04:01:34.870859 32045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9 [sys.catalog]: This master's current role is: LEADER
I20260927 04:01:34.871492 32052 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:01:34.872313 32052 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:01:34.872442 32052 catalog_manager.cc:1259] Loaded cluster ID: b1270b1b787d43b28a6652a59d51a2bc
I20260927 04:01:34.872509 32052 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:01:34.873337 32052 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:01:34.873538 32052 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9: Loaded TSK: 0
I20260927 04:01:34.880751 32052 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9: Generated new TSK 1
I20260927 04:01:34.880856 32052 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 04:01:34.956738 32043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:01:34.956991 32043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:01:34.957031 32043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:01:34.958194 32043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:01:34.958274 32043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:01:34.958319 32043 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.65
I20260927 04:01:34.959844 32043 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:39459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab
--principal=kudu/127.15.185.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.65:39393
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.185.65
--webserver_port=40253
--tserver_master_addrs=127.15.185.126:33279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32043
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.65
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:01:34.960127 32043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:01:34.960369 32043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:01:34.966439 32067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:01:34.966463 32068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:01:34.966606 32043 server_base.cc:1034] running on GCE node
W20260927 04:01:34.966504 32070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:01:34.966894 32043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:01:34.967176 32043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:01:34.968362 32043 hybrid_clock.cc:648] HybridClock initialized: now 1790481694968335 us; error 44 us; skew 500 ppm
Sep 27 04:01:34 dist-test-slave-glgz krb5kdc[31422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481694, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:01:34.979729 32043 init.cc:377] Logged in from keytab as kudu/127.15.185.65@KRBTEST.COM (short username kudu)
I20260927 04:01:34.980302 32043 webserver.cc:460] Webserver started at http://127.15.185.65:40253/ using document root <none> and password file <none>
I20260927 04:01:34.980567 32043 fs_manager.cc:359] Metadata directory not provided
I20260927 04:01:34.980624 32043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:01:34.982071 32043 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:01:34.982801 32077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:01:34.982967 32043 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:01:34.983085 32043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-0/wal
uuid: "df1bddad33464b0891abf90e06caf1b4"
format_stamp: "Formatted at 2026-09-27 04:01:14 on dist-test-slave-glgz"
server_key: "b102ecdbbbc3ad387c09d5fc28a1fbbb"
server_key_iv: "f848ed31dbfa4c943cc0616a80c68501"
server_key_version: "encryptionkey@0"
I20260927 04:01:34.983470 32043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:01:34.996853 32043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:01:34.999289 32043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:01:34.999449 32043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:01:34.999737 32043 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:01:35.000293 32084 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 04:01:35.001771 32043 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 04:01:35.001822 32043 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 04:01:35.001860 32043 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 04:01:35.002561 32043 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 04:01:35.002605 32043 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 04:01:35.002655 32084 tablet_bootstrap.cc:492] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P df1bddad33464b0891abf90e06caf1b4: Bootstrap starting.
I20260927 04:01:35.010524 32043 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.65:39393
I20260927 04:01:35.010605 32191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.65:39393 every 8 connection(s)
I20260927 04:01:35.011323 32043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
I20260927 04:01:35.014902 32084 log.cc:824] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P df1bddad33464b0891abf90e06caf1b4: Log is configured to *not* fsync() on all Append() calls
I20260927 04:01:35.016258 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 32043
I20260927 04:01:35.016697 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.66:33351
--local_ip_for_outbound_sockets=127.15.185.66
--tserver_master_addrs=127.15.185.126:33279
--webserver_port=42655
--webserver_interface=127.15.185.66
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab
--principal=kudu/127.15.185.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:39459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 04:01:35.017724 32084 tablet_bootstrap.cc:492] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P df1bddad33464b0891abf90e06caf1b4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 04:01:35.018024 32084 tablet_bootstrap.cc:492] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P df1bddad33464b0891abf90e06caf1b4: Bootstrap complete.
I20260927 04:01:35.019371 32084 ts_tablet_manager.cc:1397] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P df1bddad33464b0891abf90e06caf1b4: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.016s
I20260927 04:01:35.021174 32084 raft_consensus.cc:348] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P df1bddad33464b0891abf90e06caf1b4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10004b9896fe4d70a89becb761545a44" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 33351 } } peers { permanent_uuid: "df1bddad33464b0891abf90e06caf1b4" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 39393 } } peers { permanent_uuid: "85dd61e73f4747529e1c38c05ae79d91" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 39469 } }
I20260927 04:01:35.021601 32084 raft_consensus.cc:729] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P df1bddad33464b0891abf90e06caf1b4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1bddad33464b0891abf90e06caf1b4, State: Initialized, Role: FOLLOWER
Sep 27 04:01:35 dist-test-slave-glgz krb5kdc[31422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481694, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.65@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 04:01:35.021801 32084 consensus_queue.cc:260] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P df1bddad33464b0891abf90e06caf1b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10004b9896fe4d70a89becb761545a44" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 33351 } } peers { permanent_uuid: "df1bddad33464b0891abf90e06caf1b4" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 39393 } } peers { permanent_uuid: "85dd61e73f4747529e1c38c05ae79d91" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 39469 } }
I20260927 04:01:35.022135 32084 ts_tablet_manager.cc:1428] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P df1bddad33464b0891abf90e06caf1b4: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 04:01:35.025904 32194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:35.012096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:33279 (local address 127.15.185.65:38243)
0927 04:01:35.012301 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:35.012346 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 04:01:35.013148 (+ 802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:01:35.015651 (+ 2503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:01:35.015656 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:01:35.016048 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:01:35.016322 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:01:35.016329 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:01:35.018000 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:01:35.018001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:01:35.018965 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:01:35.018969 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:01:35.019725 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:01:35.019727 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:01:35.019866 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:01:35.020456 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:01:35.020465 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:01:35.022138 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:01:35.024272 (+ 2134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:01:35.024279 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:01:35.024281 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:01:35.024513 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:01:35.024815 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:01:35.024816 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:01:35.024816 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:01:35.024930 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:01:35.025371 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:01:35.025377 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:01:35.025530 (+ 153us) client_negotiation.cc:769] Sending connection context
0927 04:01:35.025607 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 04:01:35.025746 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":123,"threads_started":1}
I20260927 04:01:35.026392 32195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:35.012196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:38243 (local address 127.15.185.126:33279)
0927 04:01:35.015169 (+ 2973us) server_negotiation.cc:206] Beginning negotiation
0927 04:01:35.015187 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 04:01:35.015232 (+ 45us) server_negotiation.cc:407] Connection header received
0927 04:01:35.015373 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:01:35.015380 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:01:35.015437 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:01:35.015591 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:01:35.016520 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:01:35.017856 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:01:35.019086 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:01:35.019594 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:01:35.019727 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:01:35.022270 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:01:35.022275 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:01:35.022275 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:01:35.022289 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:01:35.024161 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:01:35.024616 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:01:35.024620 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:01:35.024620 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:01:35.024737 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:01:35.024997 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:01:35.024998 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:01:35.024999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:01:35.025256 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:01:35.026055 (+ 799us) server_negotiation.cc:1035] Waiting for connection context
0927 04:01:35.026138 (+ 83us) server_negotiation.cc:299] Negotiation successful
0927 04:01:35.026264 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2914,"thread_start_us":84,"threads_started":1}
I20260927 04:01:35.027002 32192 heartbeater.cc:347] Connected to a master server at 127.15.185.126:33279
I20260927 04:01:35.027217 32192 heartbeater.cc:464] Registering TS with master...
I20260927 04:01:35.027426 32192 heartbeater.cc:511] Master 127.15.185.126:33279 requested a full tablet report, sending...
I20260927 04:01:35.028177 32004 ts_manager.cc:194] Registered new tserver with Master: df1bddad33464b0891abf90e06caf1b4 (127.15.185.65:39393)
I20260927 04:01:35.029209 32004 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.185.65@KRBTEST.COM'} at 127.15.185.65:38243
I20260927 04:01:35.030083 32192 heartbeater.cc:503] Master 127.15.185.126:33279 was elected leader, sending a full tablet report...
I20260927 04:01:35.034477 32200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:35.028825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54360 (local address 127.15.185.65:39393)
0927 04:01:35.031148 (+ 2323us) server_negotiation.cc:206] Beginning negotiation
0927 04:01:35.031153 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:01:35.031161 (+ 8us) server_negotiation.cc:407] Connection header received
0927 04:01:35.031205 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:01:35.031206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:01:35.031219 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:01:35.031286 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:01:35.031466 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:01:35.032078 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:01:35.033625 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:01:35.034068 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:01:35.034126 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:01:35.034162 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0927 04:01:35.034377 (+ 215us) server_negotiation.cc:299] Negotiation successful
0927 04:01:35.034425 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2288,"thread_start_us":100,"threads_started":1}
I20260927 04:01:35.034497 32199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:35.028756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.65:39393 (local address 127.0.0.1:54360)
0927 04:01:35.028939 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:35.028986 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 04:01:35.029054 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:01:35.031316 (+ 2262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:01:35.031320 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:01:35.031323 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:01:35.031397 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:01:35.031399 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:01:35.032174 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:01:35.032175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:01:35.033531 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:01:35.033534 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:01:35.034154 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:01:35.034155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:01:35.034260 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:01:35.034276 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 04:01:35.034354 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 04:01:35.034413 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":95,"threads_started":1}
I20260927 04:01:35.034898 32126 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f20566ff1164b67bbd8c5bd70b4c9f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:01:14 UTC) from {username='kudu', principal='kudu/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:54360
I20260927 04:01:35.035575 32201 tablet_replica.cc:330] stopping tablet replica
I20260927 04:01:35.035701 32201 raft_consensus.cc:2229] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P df1bddad33464b0891abf90e06caf1b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:01:35.035792 32201 raft_consensus.cc:2258] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P df1bddad33464b0891abf90e06caf1b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:01:35.036098 32201 ts_tablet_manager.cc:1905] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P df1bddad33464b0891abf90e06caf1b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:01:35.038224 32201 ts_tablet_manager.cc:1918] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P df1bddad33464b0891abf90e06caf1b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:01:35.038317 32201 log.cc:1196] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P df1bddad33464b0891abf90e06caf1b4: Deleting WAL directory at /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-0/wal/wals/1f20566ff1164b67bbd8c5bd70b4c9f8
I20260927 04:01:35.038533 32201 ts_tablet_manager.cc:1939] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P df1bddad33464b0891abf90e06caf1b4: Deleting consensus metadata
I20260927 04:01:35.039065 31991 catalog_manager.cc:4841] TS df1bddad33464b0891abf90e06caf1b4 (127.15.185.65:39393): tablet 1f20566ff1164b67bbd8c5bd70b4c9f8 (table test-table [id=786a1a7ffe024b07b1485d9431198551]) successfully deleted
W20260927 04:01:35.106535 32197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:01:35.106779 32197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:01:35.106804 32197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:01:35.108111 32197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:01:35.108170 32197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:01:35.108198 32197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.66
I20260927 04:01:35.109656 32197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:39459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab
--principal=kudu/127.15.185.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.66:33351
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.185.66
--webserver_port=42655
--tserver_master_addrs=127.15.185.126:33279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.66
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:01:35.109933 32197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:01:35.110123 32197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:01:35.115322 32207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:01:35.115423 32209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:01:35.115566 32197 server_base.cc:1034] running on GCE node
W20260927 04:01:35.115351 32206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:01:35.115846 32197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:01:35.116117 32197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:01:35.117298 32197 hybrid_clock.cc:648] HybridClock initialized: now 1790481695117278 us; error 42 us; skew 500 ppm
Sep 27 04:01:35 dist-test-slave-glgz krb5kdc[31422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481695, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:01:35.119956 32197 init.cc:377] Logged in from keytab as kudu/127.15.185.66@KRBTEST.COM (short username kudu)
I20260927 04:01:35.120435 32197 webserver.cc:460] Webserver started at http://127.15.185.66:42655/ using document root <none> and password file <none>
I20260927 04:01:35.120682 32197 fs_manager.cc:359] Metadata directory not provided
I20260927 04:01:35.120774 32197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:01:35.122249 32197 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 04:01:35.123167 32216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:01:35.123395 32197 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:01:35.123497 32197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-1/wal
uuid: "10004b9896fe4d70a89becb761545a44"
format_stamp: "Formatted at 2026-09-27 04:01:14 on dist-test-slave-glgz"
server_key: "3fccf75ffae8b077638b18c173b4d921"
server_key_iv: "f6b8f0c8674b473502a9526e89d2b31d"
server_key_version: "encryptionkey@0"
I20260927 04:01:35.123888 32197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:01:35.151619 32197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:01:35.153908 32197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:01:35.154085 32197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:01:35.154371 32197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:01:35.154819 32223 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 04:01:35.156507 32197 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 04:01:35.156672 32197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 04:01:35.156750 32197 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 04:01:35.157449 32197 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 04:01:35.157509 32197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 04:01:35.157615 32223 tablet_bootstrap.cc:492] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P 10004b9896fe4d70a89becb761545a44: Bootstrap starting.
I20260927 04:01:35.164903 32197 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.66:33351
I20260927 04:01:35.164973 32330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.66:33351 every 8 connection(s)
I20260927 04:01:35.165328 32197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
I20260927 04:01:35.166997 32223 log.cc:824] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P 10004b9896fe4d70a89becb761545a44: Log is configured to *not* fsync() on all Append() calls
I20260927 04:01:35.169149 32223 tablet_bootstrap.cc:492] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P 10004b9896fe4d70a89becb761545a44: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 04:01:35.169440 32223 tablet_bootstrap.cc:492] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P 10004b9896fe4d70a89becb761545a44: Bootstrap complete.
I20260927 04:01:35.170799 32223 ts_tablet_manager.cc:1397] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P 10004b9896fe4d70a89becb761545a44: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
Sep 27 04:01:35 dist-test-slave-glgz krb5kdc[31422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481695, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.66@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 04:01:35.172209 32223 raft_consensus.cc:348] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P 10004b9896fe4d70a89becb761545a44 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10004b9896fe4d70a89becb761545a44" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 33351 } } peers { permanent_uuid: "df1bddad33464b0891abf90e06caf1b4" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 39393 } } peers { permanent_uuid: "85dd61e73f4747529e1c38c05ae79d91" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 39469 } }
I20260927 04:01:35.172557 32223 raft_consensus.cc:729] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P 10004b9896fe4d70a89becb761545a44 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10004b9896fe4d70a89becb761545a44, State: Initialized, Role: FOLLOWER
I20260927 04:01:35.172731 32223 consensus_queue.cc:260] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P 10004b9896fe4d70a89becb761545a44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10004b9896fe4d70a89becb761545a44" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 33351 } } peers { permanent_uuid: "df1bddad33464b0891abf90e06caf1b4" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 39393 } } peers { permanent_uuid: "85dd61e73f4747529e1c38c05ae79d91" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 39469 } }
I20260927 04:01:35.173008 32223 ts_tablet_manager.cc:1428] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P 10004b9896fe4d70a89becb761545a44: Time spent starting tablet: real 0.002s user 0.004s sys 0.002s
I20260927 04:01:35.173375 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 32197
I20260927 04:01:35.173704 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.67:39469
--local_ip_for_outbound_sockets=127.15.185.67
--tserver_master_addrs=127.15.185.126:33279
--webserver_port=43525
--webserver_interface=127.15.185.67
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab
--principal=kudu/127.15.185.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:39459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 04:01:35.175845 32195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:35.166161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:38361 (local address 127.15.185.126:33279)
0927 04:01:35.166211 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 04:01:35.166213 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:01:35.166460 (+ 247us) server_negotiation.cc:407] Connection header received
0927 04:01:35.167020 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:01:35.167021 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:01:35.167034 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:01:35.167124 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:01:35.167783 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:01:35.168321 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:01:35.169461 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:01:35.169719 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:01:35.169809 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:01:35.172616 (+ 2807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:01:35.172678 (+ 62us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:01:35.172678 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:01:35.172686 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:01:35.174356 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:01:35.174785 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:01:35.174787 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:01:35.174787 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:01:35.174853 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:01:35.175146 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:01:35.175146 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:01:35.175147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:01:35.175266 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:01:35.175343 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 04:01:35.175614 (+ 271us) server_negotiation.cc:299] Negotiation successful
0927 04:01:35.175704 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 04:01:35.176011 32333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:35.166127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:33279 (local address 127.15.185.66:38361)
0927 04:01:35.166329 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:35.166367 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 04:01:35.166899 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:01:35.167189 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:01:35.167194 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:01:35.167474 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:01:35.167677 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:01:35.167683 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:01:35.168454 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:01:35.168455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:01:35.169368 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:01:35.169371 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:01:35.169838 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:01:35.169839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:01:35.170007 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:01:35.170694 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:01:35.170702 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:01:35.172430 (+ 1728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:01:35.174462 (+ 2032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:01:35.174469 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:01:35.174471 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:01:35.174700 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:01:35.174923 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:01:35.174924 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:01:35.174925 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:01:35.175039 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:01:35.175334 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:01:35.175341 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:01:35.175482 (+ 141us) client_negotiation.cc:769] Sending connection context
0927 04:01:35.175594 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 04:01:35.175781 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":125,"threads_started":1}
I20260927 04:01:35.176504 32331 heartbeater.cc:347] Connected to a master server at 127.15.185.126:33279
I20260927 04:01:35.176618 32331 heartbeater.cc:464] Registering TS with master...
I20260927 04:01:35.176813 32331 heartbeater.cc:511] Master 127.15.185.126:33279 requested a full tablet report, sending...
I20260927 04:01:35.177397 32004 ts_manager.cc:194] Registered new tserver with Master: 10004b9896fe4d70a89becb761545a44 (127.15.185.66:33351)
I20260927 04:01:35.177965 32004 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.185.66@KRBTEST.COM'} at 127.15.185.66:38361
I20260927 04:01:35.178737 32331 heartbeater.cc:503] Master 127.15.185.126:33279 was elected leader, sending a full tablet report...
Sep 27 04:01:35 dist-test-slave-glgz krb5kdc[31422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481694, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.126@KRBTEST.COM for kudu/127.15.185.66@KRBTEST.COM
I20260927 04:01:35.186738 32199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:35.177749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.66:33351 (local address 127.0.0.1:36156)
0927 04:01:35.177809 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:35.177820 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 04:01:35.177860 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:01:35.178151 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:01:35.178152 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:01:35.178570 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:01:35.178632 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:01:35.178634 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:01:35.180057 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:01:35.180058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:01:35.180659 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:01:35.180661 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:01:35.181354 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:01:35.181355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:01:35.181394 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:01:35.181945 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:01:35.181950 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:01:35.183586 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:01:35.185632 (+ 2046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:01:35.185641 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:01:35.185643 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:01:35.185850 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:01:35.186074 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:01:35.186075 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:01:35.186076 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:01:35.186166 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:01:35.186459 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:01:35.186465 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:01:35.186509 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 04:01:35.186563 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 04:01:35.186664 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 04:01:35.186803 32337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:35.177816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36156 (local address 127.15.185.66:33351)
0927 04:01:35.177973 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 04:01:35.177979 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 04:01:35.177990 (+ 11us) server_negotiation.cc:407] Connection header received
0927 04:01:35.178027 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:01:35.178029 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:01:35.178045 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:01:35.178724 (+ 679us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:01:35.178783 (+ 59us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:01:35.179970 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:01:35.180744 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:01:35.181273 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:01:35.181355 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:01:35.183683 (+ 2328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:01:35.183692 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:01:35.183693 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:01:35.183701 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:01:35.185543 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:01:35.185931 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:01:35.185935 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:01:35.185935 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:01:35.185997 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:01:35.186229 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:01:35.186230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:01:35.186230 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:01:35.186393 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:01:35.186478 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 04:01:35.186621 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 04:01:35.186704 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":69,"threads_started":1}
I20260927 04:01:35.187310 32265 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f20566ff1164b67bbd8c5bd70b4c9f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:01:14 UTC) from {username='kudu', principal='kudu/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:36156
I20260927 04:01:35.187901 32338 tablet_replica.cc:330] stopping tablet replica
I20260927 04:01:35.188035 32338 raft_consensus.cc:2229] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P 10004b9896fe4d70a89becb761545a44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:01:35.188109 32338 raft_consensus.cc:2258] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P 10004b9896fe4d70a89becb761545a44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:01:35.188400 32338 ts_tablet_manager.cc:1905] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P 10004b9896fe4d70a89becb761545a44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:01:35.190490 32338 ts_tablet_manager.cc:1918] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P 10004b9896fe4d70a89becb761545a44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:01:35.190572 32338 log.cc:1196] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P 10004b9896fe4d70a89becb761545a44: Deleting WAL directory at /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-1/wal/wals/1f20566ff1164b67bbd8c5bd70b4c9f8
I20260927 04:01:35.190847 32338 ts_tablet_manager.cc:1939] T 1f20566ff1164b67bbd8c5bd70b4c9f8 P 10004b9896fe4d70a89becb761545a44: Deleting consensus metadata
I20260927 04:01:35.191425 31988 catalog_manager.cc:4841] TS 10004b9896fe4d70a89becb761545a44 (127.15.185.66:33351): tablet 1f20566ff1164b67bbd8c5bd70b4c9f8 (table test-table [id=786a1a7ffe024b07b1485d9431198551]) successfully deleted
W20260927 04:01:35.259949 32336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:01:35.260205 32336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:01:35.260245 32336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:01:35.261420 32336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:01:35.261494 32336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:01:35.261539 32336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.67
I20260927 04:01:35.263017 32336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:39459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab
--principal=kudu/127.15.185.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.67:39469
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.185.67
--webserver_port=43525
--tserver_master_addrs=127.15.185.126:33279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.67
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:01:35.263317 32336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:01:35.263564 32336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:01:35.269073 32343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:01:35.269124 32336 server_base.cc:1034] running on GCE node
W20260927 04:01:35.268996 32344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:01:35.268996 32346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:01:35.269549 32336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:01:35.269869 32336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:01:35.271021 32336 hybrid_clock.cc:648] HybridClock initialized: now 1790481695271007 us; error 30 us; skew 500 ppm
Sep 27 04:01:35 dist-test-slave-glgz krb5kdc[31422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481695, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:01:35.273950 32336 init.cc:377] Logged in from keytab as kudu/127.15.185.67@KRBTEST.COM (short username kudu)
I20260927 04:01:35.274441 32336 webserver.cc:460] Webserver started at http://127.15.185.67:43525/ using document root <none> and password file <none>
I20260927 04:01:35.274701 32336 fs_manager.cc:359] Metadata directory not provided
I20260927 04:01:35.274765 32336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:01:35.276295 32336 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:01:35.277014 32353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:01:35.277209 32336 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:01:35.277324 32336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-2/wal
uuid: "85dd61e73f4747529e1c38c05ae79d91"
format_stamp: "Formatted at 2026-09-27 04:01:14 on dist-test-slave-glgz"
server_key: "d9da12f34d9db010d0142328d019cdcf"
server_key_iv: "cc190ec569713feac05a30961e495d38"
server_key_version: "encryptionkey@0"
I20260927 04:01:35.277709 32336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:01:35.301141 32336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:01:35.303365 32336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:01:35.303560 32336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:01:35.303843 32336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:01:35.304191 32336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:01:35.304250 32336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:01:35.304291 32336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:01:35.304343 32336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:01:35.310628 32336 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.67:39469
I20260927 04:01:35.310761 32466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.67:39469 every 8 connection(s)
I20260927 04:01:35.311190 32336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
Sep 27 04:01:35 dist-test-slave-glgz krb5kdc[31422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481695, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.67@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 04:01:35.319675 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 32336
I20260927 04:01:35.322053 32195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:35.312197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:44557 (local address 127.15.185.126:33279)
0927 04:01:35.312271 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0927 04:01:35.312276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:01:35.312565 (+ 289us) server_negotiation.cc:407] Connection header received
0927 04:01:35.313156 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:01:35.313157 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:01:35.313172 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:01:35.313243 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:01:35.314277 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:01:35.314782 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:01:35.315867 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:01:35.316072 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:01:35.316161 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:01:35.318555 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:01:35.318557 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:01:35.318557 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:01:35.318565 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:01:35.320383 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:01:35.320919 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:01:35.320921 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:01:35.320922 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:01:35.320993 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:01:35.321292 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:01:35.321294 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:01:35.321294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:01:35.321449 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:01:35.321534 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 04:01:35.321866 (+ 332us) server_negotiation.cc:299] Negotiation successful
0927 04:01:35.321959 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 04:01:35.322341 32469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:35.312209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:33279 (local address 127.15.185.67:44557)
0927 04:01:35.312430 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:35.312469 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 04:01:35.313037 (+ 568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:01:35.313337 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:01:35.313341 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:01:35.313958 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:01:35.314169 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:01:35.314175 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:01:35.314883 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:01:35.314885 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:01:35.315774 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:01:35.315777 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:01:35.316172 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:01:35.316174 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:01:35.316307 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:01:35.316857 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:01:35.316866 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:01:35.318425 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:01:35.320497 (+ 2072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:01:35.320506 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:01:35.320508 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:01:35.320812 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:01:35.321070 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:01:35.321071 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:01:35.321071 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:01:35.321211 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:01:35.321524 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:01:35.321532 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:01:35.321698 (+ 166us) client_negotiation.cc:769] Sending connection context
0927 04:01:35.321830 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 04:01:35.322043 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":119,"threads_started":1}
I20260927 04:01:35.322944 32467 heartbeater.cc:347] Connected to a master server at 127.15.185.126:33279
I20260927 04:01:35.323129 32467 heartbeater.cc:464] Registering TS with master...
I20260927 04:01:35.323351 32467 heartbeater.cc:511] Master 127.15.185.126:33279 requested a full tablet report, sending...
I20260927 04:01:35.323726 32004 ts_manager.cc:194] Registered new tserver with Master: 85dd61e73f4747529e1c38c05ae79d91 (127.15.185.67:39469)
I20260927 04:01:35.324090 32004 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.185.67@KRBTEST.COM'} at 127.15.185.67:44557
I20260927 04:01:35.327771 32470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:35.320146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58946 (local address 127.15.185.126:33279)
0927 04:01:35.320290 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 04:01:35.320293 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:01:35.320408 (+ 115us) server_negotiation.cc:407] Connection header received
0927 04:01:35.320490 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:01:35.320491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:01:35.320510 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:01:35.320590 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:01:35.321279 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:01:35.321898 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:01:35.323167 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:01:35.323464 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:01:35.323559 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:01:35.324932 (+ 1373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:01:35.324933 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:01:35.324934 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:01:35.324942 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:01:35.326438 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:01:35.326834 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:01:35.326836 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:01:35.326836 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:01:35.326903 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:01:35.327175 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:01:35.327176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:01:35.327176 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:01:35.327301 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:01:35.327495 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0927 04:01:35.327535 (+ 40us) server_negotiation.cc:299] Negotiation successful
0927 04:01:35.327609 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":85,"threads_started":1}
I20260927 04:01:35.328503 16101 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 04:01:35.331404 32470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:35.328937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58954 (local address 127.15.185.126:33279)
0927 04:01:35.328993 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 04:01:35.328995 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:01:35.329124 (+ 129us) server_negotiation.cc:407] Connection header received
0927 04:01:35.329198 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:01:35.329198 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:01:35.329229 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:01:35.329289 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:01:35.329409 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:01:35.329882 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:01:35.330655 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:01:35.330878 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:01:35.330929 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:01:35.331169 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:01:35.331220 (+ 51us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0927 04:01:35.331355 (+ 135us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58954: token expired
Metrics: {"server-negotiator.queue_time_us":35}
W20260927 04:01:35.331485 32470 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58954: token expired
W20260927 04:01:35.331727 31930 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.15.185.126:33279)
I20260927 04:01:35.337677 32470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:35.331941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58968 (local address 127.15.185.126:33279)
0927 04:01:35.331986 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 04:01:35.331988 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:01:35.331997 (+ 9us) server_negotiation.cc:407] Connection header received
0927 04:01:35.332054 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:01:35.332055 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:01:35.332065 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:01:35.332116 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:01:35.332602 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:01:35.333060 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:01:35.333818 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:01:35.334007 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:01:35.334090 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:01:35.335297 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:01:35.335298 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:01:35.335299 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:01:35.335305 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:01:35.336516 (+ 1211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:01:35.336896 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:01:35.336897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:01:35.336898 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:01:35.336957 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:01:35.337155 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:01:35.337156 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:01:35.337156 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:01:35.337268 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:01:35.337373 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 04:01:35.337502 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 04:01:35.337577 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 04:01:35.338783 32004 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58968:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 04:01:35.340485 32004 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 04:01:35.342581 32265 tablet_service.cc:1467] Processing CreateTablet for tablet 95fb594ffffd4f569d39abe46effa17d (DEFAULT_TABLE table=test-table [id=d5cf9faa44964078a0176aab2ec61939]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:01:35.342623 32126 tablet_service.cc:1467] Processing CreateTablet for tablet 95fb594ffffd4f569d39abe46effa17d (DEFAULT_TABLE table=test-table [id=d5cf9faa44964078a0176aab2ec61939]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:01:35.342779 32126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95fb594ffffd4f569d39abe46effa17d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:01:35.342779 32265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95fb594ffffd4f569d39abe46effa17d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:01:35.344353 32223 tablet_bootstrap.cc:492] T 95fb594ffffd4f569d39abe46effa17d P 10004b9896fe4d70a89becb761545a44: Bootstrap starting.
I20260927 04:01:35.344671 32223 tablet_bootstrap.cc:654] T 95fb594ffffd4f569d39abe46effa17d P 10004b9896fe4d70a89becb761545a44: Neither blocks nor log segments found. Creating new log.
I20260927 04:01:35.345224 32084 tablet_bootstrap.cc:492] T 95fb594ffffd4f569d39abe46effa17d P df1bddad33464b0891abf90e06caf1b4: Bootstrap starting.
I20260927 04:01:35.345573 32084 tablet_bootstrap.cc:654] T 95fb594ffffd4f569d39abe46effa17d P df1bddad33464b0891abf90e06caf1b4: Neither blocks nor log segments found. Creating new log.
I20260927 04:01:35.345983 32223 tablet_bootstrap.cc:492] T 95fb594ffffd4f569d39abe46effa17d P 10004b9896fe4d70a89becb761545a44: No bootstrap required, opened a new log
I20260927 04:01:35.346048 32223 ts_tablet_manager.cc:1397] T 95fb594ffffd4f569d39abe46effa17d P 10004b9896fe4d70a89becb761545a44: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 04:01:35.346135 32084 tablet_bootstrap.cc:492] T 95fb594ffffd4f569d39abe46effa17d P df1bddad33464b0891abf90e06caf1b4: No bootstrap required, opened a new log
I20260927 04:01:35.346181 32084 ts_tablet_manager.cc:1397] T 95fb594ffffd4f569d39abe46effa17d P df1bddad33464b0891abf90e06caf1b4: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 04:01:35.346202 32223 raft_consensus.cc:348] T 95fb594ffffd4f569d39abe46effa17d P 10004b9896fe4d70a89becb761545a44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85dd61e73f4747529e1c38c05ae79d91" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 39469 } } peers { permanent_uuid: "10004b9896fe4d70a89becb761545a44" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 33351 } } peers { permanent_uuid: "df1bddad33464b0891abf90e06caf1b4" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 39393 } }
I20260927 04:01:35.346282 32223 raft_consensus.cc:374] T 95fb594ffffd4f569d39abe46effa17d P 10004b9896fe4d70a89becb761545a44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:01:35.346297 32084 raft_consensus.cc:348] T 95fb594ffffd4f569d39abe46effa17d P df1bddad33464b0891abf90e06caf1b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85dd61e73f4747529e1c38c05ae79d91" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 39469 } } peers { permanent_uuid: "10004b9896fe4d70a89becb761545a44" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 33351 } } peers { permanent_uuid: "df1bddad33464b0891abf90e06caf1b4" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 39393 } }
I20260927 04:01:35.346339 32223 raft_consensus.cc:729] T 95fb594ffffd4f569d39abe46effa17d P 10004b9896fe4d70a89becb761545a44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10004b9896fe4d70a89becb761545a44, State: Initialized, Role: FOLLOWER
I20260927 04:01:35.346364 32084 raft_consensus.cc:374] T 95fb594ffffd4f569d39abe46effa17d P df1bddad33464b0891abf90e06caf1b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:01:35.346406 32084 raft_consensus.cc:729] T 95fb594ffffd4f569d39abe46effa17d P df1bddad33464b0891abf90e06caf1b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1bddad33464b0891abf90e06caf1b4, State: Initialized, Role: FOLLOWER
I20260927 04:01:35.346388 32223 consensus_queue.cc:260] T 95fb594ffffd4f569d39abe46effa17d P 10004b9896fe4d70a89becb761545a44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85dd61e73f4747529e1c38c05ae79d91" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 39469 } } peers { permanent_uuid: "10004b9896fe4d70a89becb761545a44" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 33351 } } peers { permanent_uuid: "df1bddad33464b0891abf90e06caf1b4" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 39393 } }
I20260927 04:01:35.346411 32199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:35.342303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.67:39469 (local address 127.0.0.1:38442)
0927 04:01:35.342360 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:35.342371 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 04:01:35.342408 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:01:35.342971 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:01:35.342972 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:01:35.342974 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:01:35.343025 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:01:35.343026 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:01:35.344096 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:01:35.344097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:01:35.345195 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:01:35.345198 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:01:35.346154 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:01:35.346155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:01:35.346223 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:01:35.346227 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 04:01:35.346296 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 04:01:35.346337 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 04:01:35.346438 32473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:35.342522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38442 (local address 127.15.185.67:39469)
0927 04:01:35.342751 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 04:01:35.342756 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:01:35.342767 (+ 11us) server_negotiation.cc:407] Connection header received
0927 04:01:35.342807 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:01:35.342809 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:01:35.342827 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:01:35.342976 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:01:35.343126 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:01:35.344009 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:01:35.345297 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:01:35.346080 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:01:35.346157 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:01:35.346229 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 04:01:35.346311 (+ 82us) server_negotiation.cc:299] Negotiation successful
0927 04:01:35.346355 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":111,"threads_started":1}
I20260927 04:01:35.346455 32084 consensus_queue.cc:260] T 95fb594ffffd4f569d39abe46effa17d P df1bddad33464b0891abf90e06caf1b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85dd61e73f4747529e1c38c05ae79d91" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 39469 } } peers { permanent_uuid: "10004b9896fe4d70a89becb761545a44" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 33351 } } peers { permanent_uuid: "df1bddad33464b0891abf90e06caf1b4" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 39393 } }
I20260927 04:01:35.346483 32223 ts_tablet_manager.cc:1428] T 95fb594ffffd4f569d39abe46effa17d P 10004b9896fe4d70a89becb761545a44: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 04:01:35.346553 32084 ts_tablet_manager.cc:1428] T 95fb594ffffd4f569d39abe46effa17d P df1bddad33464b0891abf90e06caf1b4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 04:01:35.347484 32401 tablet_service.cc:1467] Processing CreateTablet for tablet 95fb594ffffd4f569d39abe46effa17d (DEFAULT_TABLE table=test-table [id=d5cf9faa44964078a0176aab2ec61939]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:01:35.347833 32401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95fb594ffffd4f569d39abe46effa17d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:01:35.350623 32476 tablet_bootstrap.cc:492] T 95fb594ffffd4f569d39abe46effa17d P 85dd61e73f4747529e1c38c05ae79d91: Bootstrap starting.
I20260927 04:01:35.351332 32476 tablet_bootstrap.cc:654] T 95fb594ffffd4f569d39abe46effa17d P 85dd61e73f4747529e1c38c05ae79d91: Neither blocks nor log segments found. Creating new log.
I20260927 04:01:35.351759 32476 log.cc:824] T 95fb594ffffd4f569d39abe46effa17d P 85dd61e73f4747529e1c38c05ae79d91: Log is configured to *not* fsync() on all Append() calls
I20260927 04:01:35.352434 32476 tablet_bootstrap.cc:492] T 95fb594ffffd4f569d39abe46effa17d P 85dd61e73f4747529e1c38c05ae79d91: No bootstrap required, opened a new log
I20260927 04:01:35.352488 32476 ts_tablet_manager.cc:1397] T 95fb594ffffd4f569d39abe46effa17d P 85dd61e73f4747529e1c38c05ae79d91: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 04:01:35.353776 32476 raft_consensus.cc:348] T 95fb594ffffd4f569d39abe46effa17d P 85dd61e73f4747529e1c38c05ae79d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85dd61e73f4747529e1c38c05ae79d91" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 39469 } } peers { permanent_uuid: "10004b9896fe4d70a89becb761545a44" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 33351 } } peers { permanent_uuid: "df1bddad33464b0891abf90e06caf1b4" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 39393 } }
I20260927 04:01:35.353914 32476 raft_consensus.cc:374] T 95fb594ffffd4f569d39abe46effa17d P 85dd61e73f4747529e1c38c05ae79d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:01:35.353936 32476 raft_consensus.cc:729] T 95fb594ffffd4f569d39abe46effa17d P 85dd61e73f4747529e1c38c05ae79d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85dd61e73f4747529e1c38c05ae79d91, State: Initialized, Role: FOLLOWER
I20260927 04:01:35.354022 32476 consensus_queue.cc:260] T 95fb594ffffd4f569d39abe46effa17d P 85dd61e73f4747529e1c38c05ae79d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85dd61e73f4747529e1c38c05ae79d91" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 39469 } } peers { permanent_uuid: "10004b9896fe4d70a89becb761545a44" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 33351 } } peers { permanent_uuid: "df1bddad33464b0891abf90e06caf1b4" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 39393 } }
I20260927 04:01:35.354314 32476 ts_tablet_manager.cc:1428] T 95fb594ffffd4f569d39abe46effa17d P 85dd61e73f4747529e1c38c05ae79d91: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 04:01:35.354328 32467 heartbeater.cc:503] Master 127.15.185.126:33279 was elected leader, sending a full tablet report...
W20260927 04:01:35.415964 32332 tablet.cc:2369] T 95fb594ffffd4f569d39abe46effa17d P 10004b9896fe4d70a89becb761545a44: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 04:01:35.474133 32198 raft_consensus.cc:482] T 95fb594ffffd4f569d39abe46effa17d P df1bddad33464b0891abf90e06caf1b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:01:35.474295 32198 raft_consensus.cc:504] T 95fb594ffffd4f569d39abe46effa17d P df1bddad33464b0891abf90e06caf1b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85dd61e73f4747529e1c38c05ae79d91" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 39469 } } peers { permanent_uuid: "10004b9896fe4d70a89becb761545a44" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 33351 } } peers { permanent_uuid: "df1bddad33464b0891abf90e06caf1b4" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 39393 } }
I20260927 04:01:35.474726 32198 leader_election.cc:288] T 95fb594ffffd4f569d39abe46effa17d P df1bddad33464b0891abf90e06caf1b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85dd61e73f4747529e1c38c05ae79d91 (127.15.185.67:39469), 10004b9896fe4d70a89becb761545a44 (127.15.185.66:33351)
I20260927 04:01:35.479090 32337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:35.475027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:40199 (local address 127.15.185.66:33351)
0927 04:01:35.475086 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 04:01:35.475088 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:01:35.475101 (+ 13us) server_negotiation.cc:407] Connection header received
0927 04:01:35.475145 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:01:35.475146 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:01:35.475168 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:01:35.475231 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:01:35.475777 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:01:35.476483 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:01:35.478077 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:01:35.478744 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:01:35.478813 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:01:35.478875 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 04:01:35.478954 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 04:01:35.478998 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 04:01:35.479090 32473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:35.475023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:56615 (local address 127.15.185.67:39469)
0927 04:01:35.475086 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 04:01:35.475088 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:01:35.475396 (+ 308us) server_negotiation.cc:407] Connection header received
0927 04:01:35.475422 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:01:35.475423 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:01:35.475431 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:01:35.475477 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:01:35.475673 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:01:35.476457 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:01:35.478076 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:01:35.478642 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:01:35.478726 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:01:35.478735 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 04:01:35.478954 (+ 219us) server_negotiation.cc:299] Negotiation successful
0927 04:01:35.478997 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 04:01:35.479099 32194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:35.474927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.66:33351 (local address 127.15.185.65:40199)
0927 04:01:35.474992 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:35.475006 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 04:01:35.475077 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:01:35.475278 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:01:35.475279 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:01:35.475281 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:01:35.475685 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:01:35.475688 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:01:35.476562 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:01:35.476563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:01:35.477965 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:01:35.477968 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:01:35.478818 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:01:35.478819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:01:35.478881 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:01:35.478883 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 04:01:35.478935 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 04:01:35.478973 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":81}
I20260927 04:01:35.479115 32479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:35.474927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.67:39469 (local address 127.15.185.65:56615)
0927 04:01:35.475198 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:35.475207 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 04:01:35.475261 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:01:35.475510 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:01:35.475510 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:01:35.475512 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:01:35.475598 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:01:35.475601 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:01:35.476562 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:01:35.476563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:01:35.477965 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:01:35.477968 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:01:35.478772 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:01:35.478773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:01:35.478862 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:01:35.478868 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 04:01:35.478930 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 04:01:35.478973 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":158,"thread_start_us":156,"threads_started":1}
I20260927 04:01:35.479471 32285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95fb594ffffd4f569d39abe46effa17d" candidate_uuid: "df1bddad33464b0891abf90e06caf1b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10004b9896fe4d70a89becb761545a44" is_pre_election: true
I20260927 04:01:35.479626 32285 raft_consensus.cc:2454] T 95fb594ffffd4f569d39abe46effa17d P 10004b9896fe4d70a89becb761545a44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1bddad33464b0891abf90e06caf1b4 in term 0.
I20260927 04:01:35.479858 32078 leader_election.cc:302] T 95fb594ffffd4f569d39abe46effa17d P df1bddad33464b0891abf90e06caf1b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10004b9896fe4d70a89becb761545a44, df1bddad33464b0891abf90e06caf1b4; no voters:
I20260927 04:01:35.479888 32421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95fb594ffffd4f569d39abe46effa17d" candidate_uuid: "df1bddad33464b0891abf90e06caf1b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85dd61e73f4747529e1c38c05ae79d91" is_pre_election: true
I20260927 04:01:35.479983 32198 raft_consensus.cc:2781] T 95fb594ffffd4f569d39abe46effa17d P df1bddad33464b0891abf90e06caf1b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 04:01:35.480067 32198 raft_consensus.cc:482] T 95fb594ffffd4f569d39abe46effa17d P df1bddad33464b0891abf90e06caf1b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 04:01:35.480057 32421 raft_consensus.cc:2454] T 95fb594ffffd4f569d39abe46effa17d P 85dd61e73f4747529e1c38c05ae79d91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df1bddad33464b0891abf90e06caf1b4 in term 0.
I20260927 04:01:35.480100 32198 raft_consensus.cc:3037] T 95fb594ffffd4f569d39abe46effa17d P df1bddad33464b0891abf90e06caf1b4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:01:35.480949 32198 raft_consensus.cc:504] T 95fb594ffffd4f569d39abe46effa17d P df1bddad33464b0891abf90e06caf1b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85dd61e73f4747529e1c38c05ae79d91" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 39469 } } peers { permanent_uuid: "10004b9896fe4d70a89becb761545a44" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 33351 } } peers { permanent_uuid: "df1bddad33464b0891abf90e06caf1b4" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 39393 } }
I20260927 04:01:35.481097 32198 leader_election.cc:288] T 95fb594ffffd4f569d39abe46effa17d P df1bddad33464b0891abf90e06caf1b4 [CANDIDATE]: Term 1 election: Requested vote from peers 85dd61e73f4747529e1c38c05ae79d91 (127.15.185.67:39469), 10004b9896fe4d70a89becb761545a44 (127.15.185.66:33351)
I20260927 04:01:35.481273 32285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95fb594ffffd4f569d39abe46effa17d" candidate_uuid: "df1bddad33464b0891abf90e06caf1b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10004b9896fe4d70a89becb761545a44"
I20260927 04:01:35.481273 32421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95fb594ffffd4f569d39abe46effa17d" candidate_uuid: "df1bddad33464b0891abf90e06caf1b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85dd61e73f4747529e1c38c05ae79d91"
I20260927 04:01:35.481355 32285 raft_consensus.cc:3037] T 95fb594ffffd4f569d39abe46effa17d P 10004b9896fe4d70a89becb761545a44 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:01:35.481359 32421 raft_consensus.cc:3037] T 95fb594ffffd4f569d39abe46effa17d P 85dd61e73f4747529e1c38c05ae79d91 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:01:35.482210 32421 raft_consensus.cc:2454] T 95fb594ffffd4f569d39abe46effa17d P 85dd61e73f4747529e1c38c05ae79d91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1bddad33464b0891abf90e06caf1b4 in term 1.
I20260927 04:01:35.482210 32285 raft_consensus.cc:2454] T 95fb594ffffd4f569d39abe46effa17d P 10004b9896fe4d70a89becb761545a44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df1bddad33464b0891abf90e06caf1b4 in term 1.
I20260927 04:01:35.482414 32081 leader_election.cc:302] T 95fb594ffffd4f569d39abe46effa17d P df1bddad33464b0891abf90e06caf1b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85dd61e73f4747529e1c38c05ae79d91, df1bddad33464b0891abf90e06caf1b4; no voters:
I20260927 04:01:35.482504 32198 raft_consensus.cc:2781] T 95fb594ffffd4f569d39abe46effa17d P df1bddad33464b0891abf90e06caf1b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:01:35.482678 32198 raft_consensus.cc:686] T 95fb594ffffd4f569d39abe46effa17d P df1bddad33464b0891abf90e06caf1b4 [term 1 LEADER]: Becoming Leader. State: Replica: df1bddad33464b0891abf90e06caf1b4, State: Running, Role: LEADER
I20260927 04:01:35.482805 32198 consensus_queue.cc:237] T 95fb594ffffd4f569d39abe46effa17d P df1bddad33464b0891abf90e06caf1b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85dd61e73f4747529e1c38c05ae79d91" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 39469 } } peers { permanent_uuid: "10004b9896fe4d70a89becb761545a44" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 33351 } } peers { permanent_uuid: "df1bddad33464b0891abf90e06caf1b4" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 39393 } }
I20260927 04:01:35.483603 32003 catalog_manager.cc:5495] T 95fb594ffffd4f569d39abe46effa17d P df1bddad33464b0891abf90e06caf1b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to df1bddad33464b0891abf90e06caf1b4 (127.15.185.65). New cstate: current_term: 1 leader_uuid: "df1bddad33464b0891abf90e06caf1b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85dd61e73f4747529e1c38c05ae79d91" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 39469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10004b9896fe4d70a89becb761545a44" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 33351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df1bddad33464b0891abf90e06caf1b4" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 39393 } health_report { overall_health: HEALTHY } } }
I20260927 04:01:35.491153 32200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:35.488869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54372 (local address 127.15.185.65:39393)
0927 04:01:35.488912 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 04:01:35.488914 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:01:35.488922 (+ 8us) server_negotiation.cc:407] Connection header received
0927 04:01:35.488996 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:01:35.488996 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:01:35.489011 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:01:35.489067 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:01:35.489161 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:01:35.489598 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:01:35.490377 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:01:35.490551 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:01:35.490602 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:01:35.490784 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:01:35.490875 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:01:35.490929 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 04:01:35.491015 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 04:01:35.491034 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 04:01:35.492740 32285 raft_consensus.cc:1261] T 95fb594ffffd4f569d39abe46effa17d P 10004b9896fe4d70a89becb761545a44 [term 1 FOLLOWER]: Refusing update from remote peer df1bddad33464b0891abf90e06caf1b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:01:35.492790 32421 raft_consensus.cc:1261] T 95fb594ffffd4f569d39abe46effa17d P 85dd61e73f4747529e1c38c05ae79d91 [term 1 FOLLOWER]: Refusing update from remote peer df1bddad33464b0891abf90e06caf1b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:01:35.493067 32198 consensus_queue.cc:1035] T 95fb594ffffd4f569d39abe46effa17d P df1bddad33464b0891abf90e06caf1b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10004b9896fe4d70a89becb761545a44" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 33351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:01:35.493206 32480 consensus_queue.cc:1035] T 95fb594ffffd4f569d39abe46effa17d P df1bddad33464b0891abf90e06caf1b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85dd61e73f4747529e1c38c05ae79d91" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 39469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:01:35.495843 32482 mvcc.cc:205] Tried to move back new op lower bound from 7333813024735256576 to 7333813024698028032. Current Snapshot: MvccSnapshot[applied={T|T < 7333813024735256576 or (T in {7333813024735256576})}]
I20260927 04:01:35.496265 32483 mvcc.cc:205] Tried to move back new op lower bound from 7333813024735256576 to 7333813024698028032. Current Snapshot: MvccSnapshot[applied={T|T < 7333813024735256576}]
I20260927 04:01:35.497575 32004 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:01:35.497664 32004 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:01:35.498405 32004 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9: Sending DeleteTablet for 3 replicas of tablet 95fb594ffffd4f569d39abe46effa17d
I20260927 04:01:35.498742 32265 tablet_service.cc:1514] Processing DeleteTablet for tablet 95fb594ffffd4f569d39abe46effa17d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:01:35 UTC) from {username='kudu', principal='kudu/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:36156
I20260927 04:01:35.498764 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 32043
I20260927 04:01:35.498836 32338 tablet_replica.cc:330] T 95fb594ffffd4f569d39abe46effa17d P 10004b9896fe4d70a89becb761545a44: stopping tablet replica
I20260927 04:01:35.498806 32401 tablet_service.cc:1514] Processing DeleteTablet for tablet 95fb594ffffd4f569d39abe46effa17d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:01:35 UTC) from {username='kudu', principal='kudu/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:38442
I20260927 04:01:35.498878 32338 raft_consensus.cc:2229] T 95fb594ffffd4f569d39abe46effa17d P 10004b9896fe4d70a89becb761545a44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:01:35.498939 32338 raft_consensus.cc:2258] T 95fb594ffffd4f569d39abe46effa17d P 10004b9896fe4d70a89becb761545a44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:01:35.499611 32492 tablet_replica.cc:330] T 95fb594ffffd4f569d39abe46effa17d P 85dd61e73f4747529e1c38c05ae79d91: stopping tablet replica
I20260927 04:01:35.499707 32492 raft_consensus.cc:2229] T 95fb594ffffd4f569d39abe46effa17d P 85dd61e73f4747529e1c38c05ae79d91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:01:35.499805 32492 raft_consensus.cc:2258] T 95fb594ffffd4f569d39abe46effa17d P 85dd61e73f4747529e1c38c05ae79d91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:01:35.500181 32338 ts_tablet_manager.cc:1905] T 95fb594ffffd4f569d39abe46effa17d P 10004b9896fe4d70a89becb761545a44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:01:35.500249 32492 ts_tablet_manager.cc:1905] T 95fb594ffffd4f569d39abe46effa17d P 85dd61e73f4747529e1c38c05ae79d91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:01:35.501642 32338 ts_tablet_manager.cc:1918] T 95fb594ffffd4f569d39abe46effa17d P 10004b9896fe4d70a89becb761545a44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:01:35.501685 32338 log.cc:1196] T 95fb594ffffd4f569d39abe46effa17d P 10004b9896fe4d70a89becb761545a44: Deleting WAL directory at /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-1/wal/wals/95fb594ffffd4f569d39abe46effa17d
I20260927 04:01:35.501922 32338 ts_tablet_manager.cc:1939] T 95fb594ffffd4f569d39abe46effa17d P 10004b9896fe4d70a89becb761545a44: Deleting consensus metadata
I20260927 04:01:35.502166 32492 ts_tablet_manager.cc:1918] T 95fb594ffffd4f569d39abe46effa17d P 85dd61e73f4747529e1c38c05ae79d91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:01:35.502246 32492 log.cc:1196] T 95fb594ffffd4f569d39abe46effa17d P 85dd61e73f4747529e1c38c05ae79d91: Deleting WAL directory at /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-2/wal/wals/95fb594ffffd4f569d39abe46effa17d
I20260927 04:01:35.502283 31988 catalog_manager.cc:4841] TS 10004b9896fe4d70a89becb761545a44 (127.15.185.66:33351): tablet 95fb594ffffd4f569d39abe46effa17d (table test-table [id=d5cf9faa44964078a0176aab2ec61939]) successfully deleted
I20260927 04:01:35.502557 32492 ts_tablet_manager.cc:1939] T 95fb594ffffd4f569d39abe46effa17d P 85dd61e73f4747529e1c38c05ae79d91: Deleting consensus metadata
I20260927 04:01:35.503078 31991 catalog_manager.cc:4841] TS 85dd61e73f4747529e1c38c05ae79d91 (127.15.185.67:39469): tablet 95fb594ffffd4f569d39abe46effa17d (table test-table [id=d5cf9faa44964078a0176aab2ec61939]) successfully deleted
W20260927 04:01:35.505754 31991 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.185.65:39393 (error 108)
I20260927 04:01:35.506319 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 32197
I20260927 04:01:35.506335 32199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:35.506194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.65:39393 (local address 127.0.0.1:54382)
0927 04:01:35.506256 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:35.506299 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.65:39393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 04:01:35.506453 31991 catalog_manager.cc:4584] TS df1bddad33464b0891abf90e06caf1b4 (127.15.185.65:39393): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 95fb594ffffd4f569d39abe46effa17d: Network error: Client connection negotiation failed: client connection to 127.15.185.65:39393: connect: Connection refused (error 111)
I20260927 04:01:35.512439 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 32336
I20260927 04:01:35.517202 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 31971
I20260927 04:01:55.523162 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:33279
--webserver_interface=127.15.185.126
--webserver_port=45685
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:39459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.185.126:33279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:01:55.604483 32494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:01:55.604700 32494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:01:55.604732 32494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:01:55.605814 32494 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 04:01:55.605850 32494 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 04:01:55.605863 32494 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 04:01:55.605875 32494 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:01:55.605887 32494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:01:55.605901 32494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:01:55.605912 32494 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:01:55.605924 32494 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:01:55.607354 32494 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:39459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.15.185.126:33279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:33279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.185.126
--webserver_port=45685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:01:55.607550 32494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:01:55.607728 32494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:01:55.613075 32502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:01:55.613112 32499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:01:55.613085 32500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:01:55.613204 32494 server_base.cc:1034] running on GCE node
I20260927 04:01:55.613654 32494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:01:55.613974 32494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:01:55.615126 32494 hybrid_clock.cc:648] HybridClock initialized: now 1790481715615111 us; error 28 us; skew 500 ppm
Sep 27 04:01:55 dist-test-slave-glgz krb5kdc[31422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481715, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:01:55.622337 32494 init.cc:377] Logged in from keytab as kudu/127.15.185.126@KRBTEST.COM (short username kudu)
I20260927 04:01:55.622843 32494 webserver.cc:460] Webserver started at http://127.15.185.126:45685/ using document root <none> and password file <none>
I20260927 04:01:55.623117 32494 fs_manager.cc:359] Metadata directory not provided
I20260927 04:01:55.623170 32494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:01:55.624794 32494 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:01:55.625634 32509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:01:55.625792 32494 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:01:55.625886 32494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/master-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/master-0/wal
uuid: "6c975de4ec944e8e8fad12e086d292e9"
format_stamp: "Formatted at 2026-09-27 04:01:13 on dist-test-slave-glgz"
server_key: "8b1376547d4f7b1cf388fbd01c0dfe45"
server_key_iv: "e943c1a811e2b8bf79813ad9af0db01c"
server_key_version: "encryptionkey@0"
I20260927 04:01:55.626256 32494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:01:55.635779 32494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:01:55.637878 32494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:01:55.637995 32494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:01:55.641999 32494 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.126:33279
I20260927 04:01:55.642028 32561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.126:33279 every 8 connection(s)
I20260927 04:01:55.642478 32494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
I20260927 04:01:55.644076 32562 sys_catalog.cc:263] Verifying existing consensus state
I20260927 04:01:55.644778 32562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9: Bootstrap starting.
I20260927 04:01:55.648535 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 32494
I20260927 04:01:55.648850 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.65:39393
--local_ip_for_outbound_sockets=127.15.185.65
--tserver_master_addrs=127.15.185.126:33279
--webserver_port=40253
--webserver_interface=127.15.185.65
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab
--principal=kudu/127.15.185.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:39459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 04:01:55.653816 32562 log.cc:824] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9: Log is configured to *not* fsync() on all Append() calls
I20260927 04:01:55.656198 32562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 04:01:55.656509 32562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9: Bootstrap complete.
I20260927 04:01:55.658725 32562 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c975de4ec944e8e8fad12e086d292e9" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 33279 } }
I20260927 04:01:55.658970 32562 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c975de4ec944e8e8fad12e086d292e9, State: Initialized, Role: FOLLOWER
I20260927 04:01:55.659196 32562 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c975de4ec944e8e8fad12e086d292e9" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 33279 } }
I20260927 04:01:55.659296 32562 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:01:55.659349 32562 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:01:55.659397 32562 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9 [term 2 FOLLOWER]: Advancing to term 3
I20260927 04:01:55.660369 32562 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c975de4ec944e8e8fad12e086d292e9" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 33279 } }
I20260927 04:01:55.660487 32562 leader_election.cc:302] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c975de4ec944e8e8fad12e086d292e9; no voters:
I20260927 04:01:55.660696 32562 leader_election.cc:288] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 04:01:55.660732 32566 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 04:01:55.660847 32566 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9 [term 3 LEADER]: Becoming Leader. State: Replica: 6c975de4ec944e8e8fad12e086d292e9, State: Running, Role: LEADER
I20260927 04:01:55.660974 32562 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:01:55.660993 32566 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c975de4ec944e8e8fad12e086d292e9" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 33279 } }
I20260927 04:01:55.661353 32567 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "6c975de4ec944e8e8fad12e086d292e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c975de4ec944e8e8fad12e086d292e9" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 33279 } } }
I20260927 04:01:55.661429 32567 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9 [sys.catalog]: This master's current role is: LEADER
I20260927 04:01:55.661746 32568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c975de4ec944e8e8fad12e086d292e9. Latest consensus state: current_term: 3 leader_uuid: "6c975de4ec944e8e8fad12e086d292e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c975de4ec944e8e8fad12e086d292e9" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 33279 } } }
I20260927 04:01:55.661813 32568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9 [sys.catalog]: This master's current role is: LEADER
I20260927 04:01:55.661823 32573 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:01:55.662675 32573 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:01:55.662865 32573 catalog_manager.cc:1259] Loaded cluster ID: b1270b1b787d43b28a6652a59d51a2bc
I20260927 04:01:55.663086 32573 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:01:55.664268 32573 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:01:55.664438 32573 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9: Loaded TSK: 0
I20260927 04:01:55.664485 32573 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6c975de4ec944e8e8fad12e086d292e9: Loaded TSK: 1
F20260927 04:01:55.664707 32573 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fb3a96249ad google::LogMessage::Fail() at ??:0
@ 0x7fb3a962788a google::LogMessage::SendToLog() at ??:0
@ 0x7fb3a96244de google::LogMessage::Flush() at ??:0
@ 0x7fb3a9625a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb3abd91104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fb3abd8ed41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fb3b15821cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fb3b158248e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fb3b1582594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fb3b1548c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fb3b1552ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fb3ab4f20d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fb3ab4e7954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb3ac34b6db start_thread at ??:0
@ 0x7fb3a812261f clone at ??:0
W20260927 04:01:55.743139 32564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:01:55.743371 32564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:01:55.743408 32564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:01:55.744665 32564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:01:55.744735 32564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:01:55.744779 32564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.65
I20260927 04:01:55.746248 32564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:39459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab
--principal=kudu/127.15.185.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.65:39393
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.185.65
--webserver_port=40253
--tserver_master_addrs=127.15.185.126:33279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.65
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:01:55.746529 32564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:01:55.746770 32564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:01:55.751895 32604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:01:55.751901 32605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:01:55.752082 32564 server_base.cc:1034] running on GCE node
W20260927 04:01:55.752235 32607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:01:55.752444 32564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:01:55.752725 32564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:01:55.753895 32564 hybrid_clock.cc:648] HybridClock initialized: now 1790481715753884 us; error 27 us; skew 500 ppm
Sep 27 04:01:55 dist-test-slave-glgz krb5kdc[31422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481715, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:01:55.756583 32564 init.cc:377] Logged in from keytab as kudu/127.15.185.65@KRBTEST.COM (short username kudu)
I20260927 04:01:55.757098 32564 webserver.cc:460] Webserver started at http://127.15.185.65:40253/ using document root <none> and password file <none>
I20260927 04:01:55.757349 32564 fs_manager.cc:359] Metadata directory not provided
I20260927 04:01:55.757418 32564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:01:55.758770 32564 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:01:55.759507 32614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:01:55.759666 32564 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 04:01:55.759770 32564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-0/wal
uuid: "df1bddad33464b0891abf90e06caf1b4"
format_stamp: "Formatted at 2026-09-27 04:01:14 on dist-test-slave-glgz"
server_key: "b102ecdbbbc3ad387c09d5fc28a1fbbb"
server_key_iv: "f848ed31dbfa4c943cc0616a80c68501"
server_key_version: "encryptionkey@0"
I20260927 04:01:55.760126 32564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:01:55.772562 32564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:01:55.774941 32564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:01:55.775149 32564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:01:55.775383 32564 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:01:55.775833 32621 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 04:01:55.777823 32564 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 04:01:55.777900 32564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 04:01:55.777948 32564 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 04:01:55.778585 32564 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 04:01:55.778642 32564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 04:01:55.778728 32621 tablet_bootstrap.cc:492] T 95fb594ffffd4f569d39abe46effa17d P df1bddad33464b0891abf90e06caf1b4: Bootstrap starting.
I20260927 04:01:55.785936 32564 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.65:39393
I20260927 04:01:55.786324 32564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
I20260927 04:01:55.786825 32728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.65:39393 every 8 connection(s)
I20260927 04:01:55.789659 32621 log.cc:824] T 95fb594ffffd4f569d39abe46effa17d P df1bddad33464b0891abf90e06caf1b4: Log is configured to *not* fsync() on all Append() calls
I20260927 04:01:55.792418 32621 tablet_bootstrap.cc:492] T 95fb594ffffd4f569d39abe46effa17d P df1bddad33464b0891abf90e06caf1b4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 04:01:55.792742 32621 tablet_bootstrap.cc:492] T 95fb594ffffd4f569d39abe46effa17d P df1bddad33464b0891abf90e06caf1b4: Bootstrap complete.
I20260927 04:01:55.794010 32621 ts_tablet_manager.cc:1397] T 95fb594ffffd4f569d39abe46effa17d P df1bddad33464b0891abf90e06caf1b4: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20260927 04:01:55.794405 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 32564
I20260927 04:01:55.794670 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.66:33351
--local_ip_for_outbound_sockets=127.15.185.66
--tserver_master_addrs=127.15.185.126:33279
--webserver_port=42655
--webserver_interface=127.15.185.66
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab
--principal=kudu/127.15.185.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:39459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 04:01:55.795472 32621 raft_consensus.cc:348] T 95fb594ffffd4f569d39abe46effa17d P df1bddad33464b0891abf90e06caf1b4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85dd61e73f4747529e1c38c05ae79d91" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 39469 } } peers { permanent_uuid: "10004b9896fe4d70a89becb761545a44" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 33351 } } peers { permanent_uuid: "df1bddad33464b0891abf90e06caf1b4" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 39393 } }
I20260927 04:01:55.795897 32621 raft_consensus.cc:729] T 95fb594ffffd4f569d39abe46effa17d P df1bddad33464b0891abf90e06caf1b4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: df1bddad33464b0891abf90e06caf1b4, State: Initialized, Role: FOLLOWER
I20260927 04:01:55.796114 32621 consensus_queue.cc:260] T 95fb594ffffd4f569d39abe46effa17d P df1bddad33464b0891abf90e06caf1b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85dd61e73f4747529e1c38c05ae79d91" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 39469 } } peers { permanent_uuid: "10004b9896fe4d70a89becb761545a44" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 33351 } } peers { permanent_uuid: "df1bddad33464b0891abf90e06caf1b4" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 39393 } }
I20260927 04:01:55.796396 32621 ts_tablet_manager.cc:1428] T 95fb594ffffd4f569d39abe46effa17d P df1bddad33464b0891abf90e06caf1b4: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
W20260927 04:01:55.865899 32731 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:01:55.787099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:33279 (local address 127.15.185.65:33181)
0927 04:01:55.789930 (+ 2831us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:55.789982 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 04:01:55.790545 (+ 563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:01:55.865699 (+ 75154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:33279: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":2777,"thread_start_us":79,"threads_started":1}
W20260927 04:01:55.866145 32729 heartbeater.cc:644] Failed to heartbeat to 127.15.185.126:33279 (0 consecutive failures): Network error: Failed to ping master at 127.15.185.126:33279: Client connection negotiation failed: client connection to 127.15.185.126:33279: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 04:01:55.866508 32731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:55.866372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:33279 (local address 127.15.185.65:37351)
0927 04:01:55.866431 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:55.866477 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 04:01:55.866937 32731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:55.866792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:33279 (local address 127.15.185.65:52893)
0927 04:01:55.866842 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:55.866873 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 04:01:55.867127 32729 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 04:01:55.876994 32733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:01:55.877199 32733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:01:55.877223 32733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:01:55.878374 32733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:01:55.878415 32733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:01:55.878441 32733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.66
I20260927 04:01:55.879985 32733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:39459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab
--principal=kudu/127.15.185.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.66:33351
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.185.66
--webserver_port=42655
--tserver_master_addrs=127.15.185.126:33279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.66
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:01:55.880241 32733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:01:55.880481 32733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:01:55.886039 32740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:01:55.886060 32742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:01:55.886073 32733 server_base.cc:1034] running on GCE node
W20260927 04:01:55.886039 32739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:01:55.886439 32733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:01:55.886703 32733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:01:55.887866 32733 hybrid_clock.cc:648] HybridClock initialized: now 1790481715887846 us; error 56 us; skew 500 ppm
Sep 27 04:01:55 dist-test-slave-glgz krb5kdc[31422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481715, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:01:55.890393 32733 init.cc:377] Logged in from keytab as kudu/127.15.185.66@KRBTEST.COM (short username kudu)
I20260927 04:01:55.890892 32733 webserver.cc:460] Webserver started at http://127.15.185.66:42655/ using document root <none> and password file <none>
I20260927 04:01:55.891171 32733 fs_manager.cc:359] Metadata directory not provided
I20260927 04:01:55.891242 32733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:01:55.892570 32733 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:01:55.893253 32749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:01:55.893445 32733 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:01:55.893543 32733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-1/wal
uuid: "10004b9896fe4d70a89becb761545a44"
format_stamp: "Formatted at 2026-09-27 04:01:14 on dist-test-slave-glgz"
server_key: "3fccf75ffae8b077638b18c173b4d921"
server_key_iv: "f6b8f0c8674b473502a9526e89d2b31d"
server_key_version: "encryptionkey@0"
I20260927 04:01:55.893916 32733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:01:55.905356 32733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:01:55.907367 32733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:01:55.907526 32733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:01:55.907765 32733 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:01:55.908066 32733 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:01:55.908118 32733 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:01:55.908161 32733 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:01:55.908192 32733 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:01:55.914119 32733 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.66:33351
I20260927 04:01:55.914175 394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.66:33351 every 8 connection(s)
I20260927 04:01:55.914569 32733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
I20260927 04:01:55.915695 397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:55.915300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:33279 (local address 127.15.185.66:55367)
0927 04:01:55.915467 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:55.915539 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":79,"threads_started":1}
W20260927 04:01:55.916174 395 heartbeater.cc:644] Failed to heartbeat to 127.15.185.126:33279 (0 consecutive failures): Network error: Failed to ping master at 127.15.185.126:33279: Client connection negotiation failed: client connection to 127.15.185.126:33279: connect: Connection refused (error 111)
I20260927 04:01:55.916474 397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:55.916372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:33279 (local address 127.15.185.66:37521)
0927 04:01:55.916422 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:55.916453 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 04:01:55.916914 397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:55.916832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:33279 (local address 127.15.185.66:33369)
0927 04:01:55.916874 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:55.916898 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 04:01:55.917042 395 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 04:01:55.919780 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 32733
I20260927 04:01:55.920024 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.67:39469
--local_ip_for_outbound_sockets=127.15.185.67
--tserver_master_addrs=127.15.185.126:33279
--webserver_port=43525
--webserver_interface=127.15.185.67
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab
--principal=kudu/127.15.185.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:39459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:01:56.004133 398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:01:56.004356 398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:01:56.004382 398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:01:56.005615 398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:01:56.005662 398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:01:56.005690 398 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.67
I20260927 04:01:56.007148 398 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:39459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab
--principal=kudu/127.15.185.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.67:39469
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.185.67
--webserver_port=43525
--tserver_master_addrs=127.15.185.126:33279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.398
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.67
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:01:56.007349 398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:01:56.007560 398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:01:56.013091 404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:01:56.013283 398 server_base.cc:1034] running on GCE node
W20260927 04:01:56.013048 407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:01:56.013048 403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:01:56.013697 398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:01:56.013947 398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:01:56.015166 398 hybrid_clock.cc:648] HybridClock initialized: now 1790481716015135 us; error 37 us; skew 500 ppm
Sep 27 04:01:56 dist-test-slave-glgz krb5kdc[31422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481716, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:01:56.017863 398 init.cc:377] Logged in from keytab as kudu/127.15.185.67@KRBTEST.COM (short username kudu)
I20260927 04:01:56.018386 398 webserver.cc:460] Webserver started at http://127.15.185.67:43525/ using document root <none> and password file <none>
I20260927 04:01:56.018628 398 fs_manager.cc:359] Metadata directory not provided
I20260927 04:01:56.018707 398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:01:56.020197 398 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:01:56.020931 414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:01:56.021188 398 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:01:56.021296 398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-2/wal
uuid: "85dd61e73f4747529e1c38c05ae79d91"
format_stamp: "Formatted at 2026-09-27 04:01:14 on dist-test-slave-glgz"
server_key: "d9da12f34d9db010d0142328d019cdcf"
server_key_iv: "cc190ec569713feac05a30961e495d38"
server_key_version: "encryptionkey@0"
I20260927 04:01:56.021662 398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:01:56.029780 398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:01:56.032058 398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:01:56.032222 398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:01:56.032464 398 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:01:56.032841 398 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:01:56.032898 398 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:01:56.032943 398 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:01:56.032976 398 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:01:56.039095 398 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.67:39469
I20260927 04:01:56.039207 529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.67:39469 every 8 connection(s)
I20260927 04:01:56.039536 398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
I20260927 04:01:56.040709 532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:56.040263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:33279 (local address 127.15.185.67:33543)
0927 04:01:56.040423 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:56.040496 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":79,"threads_started":1}
W20260927 04:01:56.041139 530 heartbeater.cc:644] Failed to heartbeat to 127.15.185.126:33279 (0 consecutive failures): Network error: Failed to ping master at 127.15.185.126:33279: Client connection negotiation failed: client connection to 127.15.185.126:33279: connect: Connection refused (error 111)
I20260927 04:01:56.041435 532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:56.041334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:33279 (local address 127.15.185.67:40897)
0927 04:01:56.041381 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:56.041413 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 04:01:56.041826 532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:56.041709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:33279 (local address 127.15.185.67:57721)
0927 04:01:56.041774 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:56.041806 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 04:01:56.041986 530 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 04:01:56.045411 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 398
I20260927 04:01:56.867734 534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:56.867504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:33279 (local address 127.15.185.65:32887)
0927 04:01:56.867624 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:56.867667 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":62,"threads_started":1}
I20260927 04:01:56.917550 535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:56.917348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:33279 (local address 127.15.185.66:36619)
0927 04:01:56.917457 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:56.917493 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":59,"threads_started":1}
I20260927 04:01:57.042676 536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:57.042430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:33279 (local address 127.15.185.67:47485)
0927 04:01:57.042562 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:57.042615 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":62,"threads_started":1}
I20260927 04:01:57.208012 537 raft_consensus.cc:482] T 95fb594ffffd4f569d39abe46effa17d P df1bddad33464b0891abf90e06caf1b4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:01:57.208175 537 raft_consensus.cc:504] T 95fb594ffffd4f569d39abe46effa17d P df1bddad33464b0891abf90e06caf1b4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85dd61e73f4747529e1c38c05ae79d91" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 39469 } } peers { permanent_uuid: "10004b9896fe4d70a89becb761545a44" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 33351 } } peers { permanent_uuid: "df1bddad33464b0891abf90e06caf1b4" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 39393 } }
I20260927 04:01:57.208626 537 leader_election.cc:288] T 95fb594ffffd4f569d39abe46effa17d P df1bddad33464b0891abf90e06caf1b4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85dd61e73f4747529e1c38c05ae79d91 (127.15.185.67:39469), 10004b9896fe4d70a89becb761545a44 (127.15.185.66:33351)
Sep 27 04:01:57 dist-test-slave-glgz krb5kdc[31422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481715, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.65@KRBTEST.COM for kudu/127.15.185.66@KRBTEST.COM
Sep 27 04:01:57 dist-test-slave-glgz krb5kdc[31422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481715, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.65@KRBTEST.COM for kudu/127.15.185.67@KRBTEST.COM
I20260927 04:01:57.219957 540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:57.209041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:52105 (local address 127.15.185.67:39469)
0927 04:01:57.209199 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 04:01:57.209207 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 04:01:57.209234 (+ 27us) server_negotiation.cc:407] Connection header received
0927 04:01:57.209665 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:01:57.209667 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:01:57.209708 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:01:57.209871 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:01:57.210752 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:01:57.211630 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:01:57.212721 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:01:57.213117 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:01:57.213240 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:01:57.216183 (+ 2943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:01:57.216186 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:01:57.216187 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:01:57.216197 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:01:57.217973 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:01:57.218584 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:01:57.218587 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:01:57.218588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:01:57.218712 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:01:57.219009 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:01:57.219010 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:01:57.219010 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:01:57.219310 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:01:57.219400 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 04:01:57.219696 (+ 296us) server_negotiation.cc:299] Negotiation successful
0927 04:01:57.219845 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":70,"threads_started":1}
I20260927 04:01:57.219957 539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:57.208953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:33959 (local address 127.15.185.66:33351)
0927 04:01:57.209135 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 04:01:57.209149 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 04:01:57.209184 (+ 35us) server_negotiation.cc:407] Connection header received
0927 04:01:57.209660 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:01:57.209663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:01:57.209708 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:01:57.209862 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:01:57.210622 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:01:57.211618 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:01:57.212645 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:01:57.213117 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:01:57.213250 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:01:57.216173 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:01:57.216177 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:01:57.216177 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:01:57.216188 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:01:57.218191 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:01:57.218702 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:01:57.218704 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:01:57.218704 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:01:57.218832 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:01:57.219078 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:01:57.219079 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:01:57.219080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:01:57.219310 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:01:57.219399 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 04:01:57.219683 (+ 284us) server_negotiation.cc:299] Negotiation successful
0927 04:01:57.219819 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":59,"threads_started":1}
I20260927 04:01:57.220075 538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:57.208868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.67:39469 (local address 127.15.185.65:52105)
0927 04:01:57.209023 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:57.209034 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 04:01:57.209091 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:01:57.209902 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:01:57.209903 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:01:57.210329 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:01:57.210495 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:01:57.210500 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:01:57.211760 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:01:57.211760 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:01:57.212551 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:01:57.212639 (+ 88us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:01:57.213218 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:01:57.213219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:01:57.213332 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:01:57.213875 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:01:57.213899 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:01:57.216087 (+ 2188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:01:57.218066 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:01:57.218074 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:01:57.218076 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:01:57.218404 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:01:57.218793 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:01:57.218794 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:01:57.218794 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:01:57.218933 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:01:57.219393 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:01:57.219399 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:01:57.219515 (+ 116us) client_negotiation.cc:769] Sending connection context
0927 04:01:57.219634 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 04:01:57.219907 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":243,"thread_start_us":59,"threads_started":1}
I20260927 04:01:57.220002 534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:57.208840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.66:33351 (local address 127.15.185.65:33959)
0927 04:01:57.208903 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:57.208917 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 04:01:57.208981 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:01:57.209901 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:01:57.209903 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:01:57.210309 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:01:57.210495 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:01:57.210500 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:01:57.211738 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:01:57.211740 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:01:57.212553 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:01:57.212555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:01:57.213230 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:01:57.213231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:01:57.213349 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:01:57.214006 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:01:57.214011 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:01:57.215769 (+ 1758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:01:57.218292 (+ 2523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:01:57.218292 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:01:57.218293 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:01:57.218622 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:01:57.218906 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:01:57.218907 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:01:57.218907 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:01:57.218959 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:01:57.219394 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:01:57.219401 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:01:57.219515 (+ 114us) client_negotiation.cc:769] Sending connection context
0927 04:01:57.219628 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 04:01:57.219827 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":5}
W20260927 04:01:57.221022 32618 leader_election.cc:341] T 95fb594ffffd4f569d39abe46effa17d P df1bddad33464b0891abf90e06caf1b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 85dd61e73f4747529e1c38c05ae79d91 (127.15.185.67:39469): Not found: Tablet not found: 95fb594ffffd4f569d39abe46effa17d
W20260927 04:01:57.221179 32615 leader_election.cc:341] T 95fb594ffffd4f569d39abe46effa17d P df1bddad33464b0891abf90e06caf1b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 10004b9896fe4d70a89becb761545a44 (127.15.185.66:33351): Not found: Tablet not found: 95fb594ffffd4f569d39abe46effa17d
I20260927 04:01:57.221241 32615 leader_election.cc:302] T 95fb594ffffd4f569d39abe46effa17d P df1bddad33464b0891abf90e06caf1b4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1bddad33464b0891abf90e06caf1b4; no voters: 10004b9896fe4d70a89becb761545a44, 85dd61e73f4747529e1c38c05ae79d91
I20260927 04:01:57.221364 537 raft_consensus.cc:2726] T 95fb594ffffd4f569d39abe46effa17d P df1bddad33464b0891abf90e06caf1b4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 04:01:57.868489 541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:57.868243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:33279 (local address 127.15.185.65:55965)
0927 04:01:57.868393 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:57.868439 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":63,"threads_started":1}
I20260927 04:01:57.918332 542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:57.918083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:33279 (local address 127.15.185.66:52965)
0927 04:01:57.918242 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:57.918288 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":67,"threads_started":1}
I20260927 04:01:58.043486 543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:58.043249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:33279 (local address 127.15.185.67:54585)
0927 04:01:58.043393 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:58.043435 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":62,"threads_started":1}
I20260927 04:01:58.815903 544 raft_consensus.cc:482] T 95fb594ffffd4f569d39abe46effa17d P df1bddad33464b0891abf90e06caf1b4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:01:58.816001 544 raft_consensus.cc:504] T 95fb594ffffd4f569d39abe46effa17d P df1bddad33464b0891abf90e06caf1b4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85dd61e73f4747529e1c38c05ae79d91" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 39469 } } peers { permanent_uuid: "10004b9896fe4d70a89becb761545a44" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 33351 } } peers { permanent_uuid: "df1bddad33464b0891abf90e06caf1b4" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 39393 } }
I20260927 04:01:58.816195 544 leader_election.cc:288] T 95fb594ffffd4f569d39abe46effa17d P df1bddad33464b0891abf90e06caf1b4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85dd61e73f4747529e1c38c05ae79d91 (127.15.185.67:39469), 10004b9896fe4d70a89becb761545a44 (127.15.185.66:33351)
W20260927 04:01:58.816619 32618 leader_election.cc:341] T 95fb594ffffd4f569d39abe46effa17d P df1bddad33464b0891abf90e06caf1b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 85dd61e73f4747529e1c38c05ae79d91 (127.15.185.67:39469): Not found: Tablet not found: 95fb594ffffd4f569d39abe46effa17d
W20260927 04:01:58.816702 32615 leader_election.cc:341] T 95fb594ffffd4f569d39abe46effa17d P df1bddad33464b0891abf90e06caf1b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 10004b9896fe4d70a89becb761545a44 (127.15.185.66:33351): Not found: Tablet not found: 95fb594ffffd4f569d39abe46effa17d
I20260927 04:01:58.816802 32615 leader_election.cc:302] T 95fb594ffffd4f569d39abe46effa17d P df1bddad33464b0891abf90e06caf1b4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1bddad33464b0891abf90e06caf1b4; no voters: 10004b9896fe4d70a89becb761545a44, 85dd61e73f4747529e1c38c05ae79d91
I20260927 04:01:58.816871 544 raft_consensus.cc:2726] T 95fb594ffffd4f569d39abe46effa17d P df1bddad33464b0891abf90e06caf1b4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 04:01:58.869222 545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:58.868984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:33279 (local address 127.15.185.65:39141)
0927 04:01:58.869131 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:58.869173 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":78,"threads_started":1}
I20260927 04:01:58.919152 546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:58.918866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:33279 (local address 127.15.185.66:35441)
0927 04:01:58.918996 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:58.919036 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":61,"threads_started":1}
I20260927 04:01:59.044382 547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:59.044116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:33279 (local address 127.15.185.67:56313)
0927 04:01:59.044282 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:59.044333 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":96,"threads_started":1}
I20260927 04:01:59.870123 548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:59.869839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:33279 (local address 127.15.185.65:50163)
0927 04:01:59.869993 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:59.870040 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":67,"threads_started":1}
I20260927 04:01:59.919917 549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:01:59.919658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:33279 (local address 127.15.185.66:55937)
0927 04:01:59.919822 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 04:01:59.919866 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":98,"threads_started":1}
I20260927 04:02:00.045291 550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:00.045026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:33279 (local address 127.15.185.67:52879)
0927 04:02:00.045191 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:00.045241 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":66,"threads_started":1}
I20260927 04:02:00.870891 551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:00.870650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:33279 (local address 127.15.185.65:57309)
0927 04:02:00.870802 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:00.870844 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":74,"threads_started":1}
I20260927 04:02:00.920662 552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:00.920426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:33279 (local address 127.15.185.66:54667)
0927 04:02:00.920566 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:00.920607 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":89,"threads_started":1}
I20260927 04:02:01.046159 553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:01.045879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:33279 (local address 127.15.185.67:43591)
0927 04:02:01.046055 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:01.046111 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":79,"threads_started":1}
I20260927 04:02:01.382162 554 raft_consensus.cc:482] T 95fb594ffffd4f569d39abe46effa17d P df1bddad33464b0891abf90e06caf1b4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:02:01.382268 554 raft_consensus.cc:504] T 95fb594ffffd4f569d39abe46effa17d P df1bddad33464b0891abf90e06caf1b4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85dd61e73f4747529e1c38c05ae79d91" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 39469 } } peers { permanent_uuid: "10004b9896fe4d70a89becb761545a44" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 33351 } } peers { permanent_uuid: "df1bddad33464b0891abf90e06caf1b4" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 39393 } }
I20260927 04:02:01.382447 554 leader_election.cc:288] T 95fb594ffffd4f569d39abe46effa17d P df1bddad33464b0891abf90e06caf1b4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85dd61e73f4747529e1c38c05ae79d91 (127.15.185.67:39469), 10004b9896fe4d70a89becb761545a44 (127.15.185.66:33351)
W20260927 04:02:01.382833 32618 leader_election.cc:341] T 95fb594ffffd4f569d39abe46effa17d P df1bddad33464b0891abf90e06caf1b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 85dd61e73f4747529e1c38c05ae79d91 (127.15.185.67:39469): Not found: Tablet not found: 95fb594ffffd4f569d39abe46effa17d
W20260927 04:02:01.382918 32615 leader_election.cc:341] T 95fb594ffffd4f569d39abe46effa17d P df1bddad33464b0891abf90e06caf1b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 10004b9896fe4d70a89becb761545a44 (127.15.185.66:33351): Not found: Tablet not found: 95fb594ffffd4f569d39abe46effa17d
I20260927 04:02:01.382998 32615 leader_election.cc:302] T 95fb594ffffd4f569d39abe46effa17d P df1bddad33464b0891abf90e06caf1b4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1bddad33464b0891abf90e06caf1b4; no voters: 10004b9896fe4d70a89becb761545a44, 85dd61e73f4747529e1c38c05ae79d91
I20260927 04:02:01.383086 554 raft_consensus.cc:2726] T 95fb594ffffd4f569d39abe46effa17d P df1bddad33464b0891abf90e06caf1b4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 04:02:01.871634 555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:01.871403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:33279 (local address 127.15.185.65:52235)
0927 04:02:01.871538 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:01.871586 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":65,"threads_started":1}
I20260927 04:02:01.921377 556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:01.921118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:33279 (local address 127.15.185.66:34953)
0927 04:02:01.921286 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:01.921329 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":64,"threads_started":1}
I20260927 04:02:02.046962 557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:02.046733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:33279 (local address 127.15.185.67:50461)
0927 04:02:02.046872 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:02.046913 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":62,"threads_started":1}
I20260927 04:02:02.872400 558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:02.872156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:33279 (local address 127.15.185.65:33213)
0927 04:02:02.872302 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:02.872351 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":71,"threads_started":1}
I20260927 04:02:02.922219 559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:02.921947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:33279 (local address 127.15.185.66:51005)
0927 04:02:02.922112 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:02.922168 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":64,"threads_started":1}
I20260927 04:02:03.047735 560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:03.047498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:33279 (local address 127.15.185.67:48665)
0927 04:02:03.047645 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:03.047688 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":83,"threads_started":1}
I20260927 04:02:03.195333 561 raft_consensus.cc:482] T 95fb594ffffd4f569d39abe46effa17d P df1bddad33464b0891abf90e06caf1b4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:02:03.195430 561 raft_consensus.cc:504] T 95fb594ffffd4f569d39abe46effa17d P df1bddad33464b0891abf90e06caf1b4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85dd61e73f4747529e1c38c05ae79d91" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 39469 } } peers { permanent_uuid: "10004b9896fe4d70a89becb761545a44" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 33351 } } peers { permanent_uuid: "df1bddad33464b0891abf90e06caf1b4" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 39393 } }
I20260927 04:02:03.195585 561 leader_election.cc:288] T 95fb594ffffd4f569d39abe46effa17d P df1bddad33464b0891abf90e06caf1b4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85dd61e73f4747529e1c38c05ae79d91 (127.15.185.67:39469), 10004b9896fe4d70a89becb761545a44 (127.15.185.66:33351)
W20260927 04:02:03.196033 32615 leader_election.cc:341] T 95fb594ffffd4f569d39abe46effa17d P df1bddad33464b0891abf90e06caf1b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 10004b9896fe4d70a89becb761545a44 (127.15.185.66:33351): Not found: Tablet not found: 95fb594ffffd4f569d39abe46effa17d
W20260927 04:02:03.196149 32618 leader_election.cc:341] T 95fb594ffffd4f569d39abe46effa17d P df1bddad33464b0891abf90e06caf1b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 85dd61e73f4747529e1c38c05ae79d91 (127.15.185.67:39469): Not found: Tablet not found: 95fb594ffffd4f569d39abe46effa17d
I20260927 04:02:03.196194 32618 leader_election.cc:302] T 95fb594ffffd4f569d39abe46effa17d P df1bddad33464b0891abf90e06caf1b4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1bddad33464b0891abf90e06caf1b4; no voters: 10004b9896fe4d70a89becb761545a44, 85dd61e73f4747529e1c38c05ae79d91
I20260927 04:02:03.196269 561 raft_consensus.cc:2726] T 95fb594ffffd4f569d39abe46effa17d P df1bddad33464b0891abf90e06caf1b4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 04:02:03.873191 562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:03.872941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:33279 (local address 127.15.185.65:44813)
0927 04:02:03.873102 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:03.873141 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":92,"threads_started":1}
I20260927 04:02:03.922974 563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:03.922716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:33279 (local address 127.15.185.66:51835)
0927 04:02:03.922886 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:03.922928 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":87,"threads_started":1}
I20260927 04:02:04.048550 564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:04.048307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:33279 (local address 127.15.185.67:45449)
0927 04:02:04.048455 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:04.048502 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":77,"threads_started":1}
I20260927 04:02:04.874017 565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:04.873775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:33279 (local address 127.15.185.65:38439)
0927 04:02:04.873921 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:04.873966 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":97,"threads_started":1}
I20260927 04:02:04.923784 566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:04.923543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:33279 (local address 127.15.185.66:44093)
0927 04:02:04.923670 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:04.923715 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":60,"threads_started":1}
I20260927 04:02:05.049388 567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:05.049133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:33279 (local address 127.15.185.67:35263)
0927 04:02:05.049295 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:05.049339 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":91,"threads_started":1}
I20260927 04:02:05.874907 568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:05.874596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:33279 (local address 127.15.185.65:39667)
0927 04:02:05.874805 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:05.874855 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":79,"threads_started":1}
I20260927 04:02:05.924616 569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:05.924318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:33279 (local address 127.15.185.66:50381)
0927 04:02:05.924516 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:05.924567 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":75,"threads_started":1}
I20260927 04:02:06.050341 570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:06.050074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:33279 (local address 127.15.185.67:55047)
0927 04:02:06.050230 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:06.050273 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":101,"threads_started":1}
I20260927 04:02:06.875836 571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:06.875571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:33279 (local address 127.15.185.65:46197)
0927 04:02:06.875709 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:06.875782 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":88,"threads_started":1}
I20260927 04:02:06.925472 572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:06.925204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:33279 (local address 127.15.185.66:41389)
0927 04:02:06.925372 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:06.925421 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":70,"threads_started":1}
I20260927 04:02:07.051306 573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:07.050946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:33279 (local address 127.15.185.67:49249)
0927 04:02:07.051174 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:07.051232 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":84,"threads_started":1}
I20260927 04:02:07.710322 574 raft_consensus.cc:482] T 95fb594ffffd4f569d39abe46effa17d P df1bddad33464b0891abf90e06caf1b4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:02:07.710421 574 raft_consensus.cc:504] T 95fb594ffffd4f569d39abe46effa17d P df1bddad33464b0891abf90e06caf1b4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85dd61e73f4747529e1c38c05ae79d91" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 39469 } } peers { permanent_uuid: "10004b9896fe4d70a89becb761545a44" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 33351 } } peers { permanent_uuid: "df1bddad33464b0891abf90e06caf1b4" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 39393 } }
I20260927 04:02:07.710673 574 leader_election.cc:288] T 95fb594ffffd4f569d39abe46effa17d P df1bddad33464b0891abf90e06caf1b4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 85dd61e73f4747529e1c38c05ae79d91 (127.15.185.67:39469), 10004b9896fe4d70a89becb761545a44 (127.15.185.66:33351)
W20260927 04:02:07.711117 32615 leader_election.cc:341] T 95fb594ffffd4f569d39abe46effa17d P df1bddad33464b0891abf90e06caf1b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 10004b9896fe4d70a89becb761545a44 (127.15.185.66:33351): Not found: Tablet not found: 95fb594ffffd4f569d39abe46effa17d
W20260927 04:02:07.711208 32618 leader_election.cc:341] T 95fb594ffffd4f569d39abe46effa17d P df1bddad33464b0891abf90e06caf1b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 85dd61e73f4747529e1c38c05ae79d91 (127.15.185.67:39469): Not found: Tablet not found: 95fb594ffffd4f569d39abe46effa17d
I20260927 04:02:07.711308 32618 leader_election.cc:302] T 95fb594ffffd4f569d39abe46effa17d P df1bddad33464b0891abf90e06caf1b4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df1bddad33464b0891abf90e06caf1b4; no voters: 10004b9896fe4d70a89becb761545a44, 85dd61e73f4747529e1c38c05ae79d91
I20260927 04:02:07.711404 574 raft_consensus.cc:2726] T 95fb594ffffd4f569d39abe46effa17d P df1bddad33464b0891abf90e06caf1b4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 04:02:07.876495 575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:07.876266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:33279 (local address 127.15.185.65:55165)
0927 04:02:07.876394 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:07.876430 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":67,"threads_started":1}
I20260927 04:02:07.926301 576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:07.926067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:33279 (local address 127.15.185.66:53873)
0927 04:02:07.926193 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:07.926240 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":61,"threads_started":1}
I20260927 04:02:08.052192 577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:08.051932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:33279 (local address 127.15.185.67:44983)
0927 04:02:08.052092 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:08.052144 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":64,"threads_started":1}
I20260927 04:02:08.877441 578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:08.877068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:33279 (local address 127.15.185.65:43871)
0927 04:02:08.877317 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:08.877358 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":107,"threads_started":1}
I20260927 04:02:08.927139 579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:08.926815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:33279 (local address 127.15.185.66:47057)
0927 04:02:08.926963 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:08.927009 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":61,"threads_started":1}
I20260927 04:02:09.053030 580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:09.052744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:33279 (local address 127.15.185.67:56597)
0927 04:02:09.052935 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:09.052981 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":94,"threads_started":1}
I20260927 04:02:09.878324 581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:09.878047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:33279 (local address 127.15.185.65:45393)
0927 04:02:09.878223 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:09.878273 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":79,"threads_started":1}
I20260927 04:02:09.927863 582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:09.927637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:33279 (local address 127.15.185.66:34989)
0927 04:02:09.927770 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:09.927814 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":61,"threads_started":1}
I20260927 04:02:10.053855 583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:10.053627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:33279 (local address 127.15.185.67:53491)
0927 04:02:10.053765 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:10.053807 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":71,"threads_started":1}
I20260927 04:02:10.879287 584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:10.878997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:33279 (local address 127.15.185.65:41951)
0927 04:02:10.879189 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:10.879239 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":90,"threads_started":1}
I20260927 04:02:10.928716 585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:10.928471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:33279 (local address 127.15.185.66:47549)
0927 04:02:10.928619 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:10.928665 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":75,"threads_started":1}
I20260927 04:02:11.049937 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 32564
I20260927 04:02:11.054894 586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:11.054591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:33279 (local address 127.15.185.67:51233)
0927 04:02:11.054758 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:11.054837 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.126:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":76,"threads_started":1}
I20260927 04:02:11.055217 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 32733
I20260927 04:02:11.059703 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 398
2026-09-27T04:02:11Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58352 ms)
[----------] 30 tests from SecurityITest (233049 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:02:11 dist-test-slave-glgz krb5kdc[589](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:02:11 dist-test-slave-glgz krb5kdc[589](info): set up 2 sockets
Sep 27 04:02:11 dist-test-slave-glgz krb5kdc[589](info): commencing operation
krb5kdc: starting...
W20260927 04:02:12.115743 16101 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Sep 27 04:02:12 dist-test-slave-glgz krb5kdc[589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:02:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:02:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.185.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab.
Entry for principal kudu/127.15.185.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab.
I20260927 04:02:12.241552 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:36371
--webserver_interface=127.15.185.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:46511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.185.126:36371
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:02:12.327867 602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:02:12.328114 602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:02:12.328151 602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:02:12.329311 602 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:02:12.329381 602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:02:12.329404 602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:02:12.329428 602 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:02:12.329456 602 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:02:12.330936 602 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:46511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.185.126:36371
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:36371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:02:12.331255 602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:02:12.331489 602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:02:12.336930 608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:12.336921 610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:12.336952 602 server_base.cc:1034] running on GCE node
W20260927 04:02:12.336921 607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:12.337375 602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:02:12.337635 602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:02:12.338779 602 hybrid_clock.cc:648] HybridClock initialized: now 1790481732338763 us; error 27 us; skew 500 ppm
Sep 27 04:02:12 dist-test-slave-glgz krb5kdc[589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481732, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:02:12.341306 602 init.cc:377] Logged in from keytab as kudu/127.15.185.126@KRBTEST.COM (short username kudu)
I20260927 04:02:12.341754 602 webserver.cc:460] Webserver started at http://127.15.185.126:42341/ using document root <none> and password file <none>
I20260927 04:02:12.341948 602 fs_manager.cc:359] Metadata directory not provided
I20260927 04:02:12.341990 602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:02:12.342087 602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:02:12.343163 602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/master-0/data/instance:
uuid: "bb208bd43f804dc085b4d803a6755f74"
format_stamp: "Formatted at 2026-09-27 04:02:12 on dist-test-slave-glgz"
server_key: "039cd43bcf3660eac2a5b7b77e9286c8"
server_key_iv: "91e7a25dc67800c000aa1254a7996d60"
server_key_version: "encryptionkey@0"
I20260927 04:02:12.343482 602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/master-0/wal/instance:
uuid: "bb208bd43f804dc085b4d803a6755f74"
format_stamp: "Formatted at 2026-09-27 04:02:12 on dist-test-slave-glgz"
server_key: "039cd43bcf3660eac2a5b7b77e9286c8"
server_key_iv: "91e7a25dc67800c000aa1254a7996d60"
server_key_version: "encryptionkey@0"
I20260927 04:02:12.344789 602 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:02:12.345494 617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:12.345674 602 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 04:02:12.345767 602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/master-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/master-0/wal
uuid: "bb208bd43f804dc085b4d803a6755f74"
format_stamp: "Formatted at 2026-09-27 04:02:12 on dist-test-slave-glgz"
server_key: "039cd43bcf3660eac2a5b7b77e9286c8"
server_key_iv: "91e7a25dc67800c000aa1254a7996d60"
server_key_version: "encryptionkey@0"
I20260927 04:02:12.345839 602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:02:12.358429 602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:02:12.360821 602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:02:12.360977 602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:02:12.364887 602 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.126:36371
I20260927 04:02:12.364951 670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.126:36371 every 8 connection(s)
I20260927 04:02:12.365298 602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
I20260927 04:02:12.366216 671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:02:12.366834 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 602
I20260927 04:02:12.366977 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/master-0/wal/instance
I20260927 04:02:12.367216 16101 external_mini_cluster.cc:1428] Setting key 29b6fe11e51c4ac0e88f9d9d54b8ace2
I20260927 04:02:12.369350 671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb208bd43f804dc085b4d803a6755f74: Bootstrap starting.
I20260927 04:02:12.370029 671 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb208bd43f804dc085b4d803a6755f74: Neither blocks nor log segments found. Creating new log.
I20260927 04:02:12.370359 671 log.cc:824] T 00000000000000000000000000000000 P bb208bd43f804dc085b4d803a6755f74: Log is configured to *not* fsync() on all Append() calls
I20260927 04:02:12.371097 671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb208bd43f804dc085b4d803a6755f74: No bootstrap required, opened a new log
I20260927 04:02:12.373257 671 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb208bd43f804dc085b4d803a6755f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb208bd43f804dc085b4d803a6755f74" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 36371 } }
I20260927 04:02:12.373390 671 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb208bd43f804dc085b4d803a6755f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:02:12.373413 671 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb208bd43f804dc085b4d803a6755f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb208bd43f804dc085b4d803a6755f74, State: Initialized, Role: FOLLOWER
I20260927 04:02:12.373562 671 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb208bd43f804dc085b4d803a6755f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb208bd43f804dc085b4d803a6755f74" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 36371 } }
I20260927 04:02:12.373641 671 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb208bd43f804dc085b4d803a6755f74 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:02:12.373663 671 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb208bd43f804dc085b4d803a6755f74 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:02:12.373700 671 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb208bd43f804dc085b4d803a6755f74 [term 0 FOLLOWER]: Advancing to term 1
Sep 27 04:02:12 dist-test-slave-glgz krb5kdc[589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 04:02:12.374548 671 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb208bd43f804dc085b4d803a6755f74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb208bd43f804dc085b4d803a6755f74" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 36371 } }
I20260927 04:02:12.374696 671 leader_election.cc:302] T 00000000000000000000000000000000 P bb208bd43f804dc085b4d803a6755f74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb208bd43f804dc085b4d803a6755f74; no voters:
I20260927 04:02:12.374893 671 leader_election.cc:288] T 00000000000000000000000000000000 P bb208bd43f804dc085b4d803a6755f74 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:02:12.374926 676 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb208bd43f804dc085b4d803a6755f74 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:02:12.375059 676 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb208bd43f804dc085b4d803a6755f74 [term 1 LEADER]: Becoming Leader. State: Replica: bb208bd43f804dc085b4d803a6755f74, State: Running, Role: LEADER
I20260927 04:02:12.375200 676 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb208bd43f804dc085b4d803a6755f74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb208bd43f804dc085b4d803a6755f74" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 36371 } }
I20260927 04:02:12.375186 671 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb208bd43f804dc085b4d803a6755f74 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:02:12.375636 677 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb208bd43f804dc085b4d803a6755f74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb208bd43f804dc085b4d803a6755f74. Latest consensus state: current_term: 1 leader_uuid: "bb208bd43f804dc085b4d803a6755f74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb208bd43f804dc085b4d803a6755f74" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 36371 } } }
I20260927 04:02:12.375710 677 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb208bd43f804dc085b4d803a6755f74 [sys.catalog]: This master's current role is: LEADER
I20260927 04:02:12.375934 678 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb208bd43f804dc085b4d803a6755f74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb208bd43f804dc085b4d803a6755f74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb208bd43f804dc085b4d803a6755f74" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 36371 } } }
I20260927 04:02:12.376052 678 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb208bd43f804dc085b4d803a6755f74 [sys.catalog]: This master's current role is: LEADER
I20260927 04:02:12.376202 681 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:02:12.376735 681 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:02:12.378371 681 catalog_manager.cc:1347] Generated new cluster ID: 68dc15ecacdf4c42bf2c97f739376ec0
I20260927 04:02:12.378417 681 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:02:12.378842 674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:12.367781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56546 (local address 127.15.185.126:36371)
0927 04:02:12.367973 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:12.367983 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:12.368016 (+ 33us) server_negotiation.cc:407] Connection header received
0927 04:02:12.368555 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:12.368565 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:12.368629 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:12.368801 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:12.369388 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:12.370765 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:12.371605 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:12.372228 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:12.372339 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:12.374779 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:12.374783 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:12.374783 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:12.374796 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:12.377067 (+ 2271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:12.377696 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:12.377698 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:12.377699 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:12.377777 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:12.378024 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:12.378025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:12.378025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:12.378258 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:12.378474 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:12.378547 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 04:02:12.378684 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":79,"threads_started":1}
I20260927 04:02:12.398305 681 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:02:12.399103 681 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:02:12.403115 681 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb208bd43f804dc085b4d803a6755f74: Generated new TSK 0
I20260927 04:02:12.403318 681 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.185.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab.
Entry for principal kudu/127.15.185.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab.
I20260927 04:02:12.435201 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.65:0
--local_ip_for_outbound_sockets=127.15.185.65
--webserver_interface=127.15.185.65
--webserver_port=0
--tserver_master_addrs=127.15.185.126:36371
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab
--principal=kudu/127.15.185.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:46511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:02:12.517030 697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:02:12.517233 697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:02:12.517262 697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:02:12.518324 697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:02:12.518355 697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:02:12.518380 697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.65
I20260927 04:02:12.519968 697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:46511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab
--principal=kudu/127.15.185.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.185.65
--webserver_port=0
--tserver_master_addrs=127.15.185.126:36371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.65
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:02:12.520157 697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:02:12.520339 697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:02:12.525358 703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:12.525432 705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:12.525367 702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:12.525650 697 server_base.cc:1034] running on GCE node
I20260927 04:02:12.525840 697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:02:12.526031 697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:02:12.527165 697 hybrid_clock.cc:648] HybridClock initialized: now 1790481732527149 us; error 29 us; skew 500 ppm
Sep 27 04:02:12 dist-test-slave-glgz krb5kdc[589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481732, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:02:12.529709 697 init.cc:377] Logged in from keytab as kudu/127.15.185.65@KRBTEST.COM (short username kudu)
I20260927 04:02:12.530133 697 webserver.cc:460] Webserver started at http://127.15.185.65:39285/ using document root <none> and password file <none>
I20260927 04:02:12.530320 697 fs_manager.cc:359] Metadata directory not provided
I20260927 04:02:12.530361 697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:02:12.530447 697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:02:12.531534 697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/ts-0/data/instance:
uuid: "f6c26349bd834fe8b3f4714e2f2203ea"
format_stamp: "Formatted at 2026-09-27 04:02:12 on dist-test-slave-glgz"
server_key: "158292456155e792693e957f272f133e"
server_key_iv: "89b6d2c1135e19d4af44854e5942bcdb"
server_key_version: "encryptionkey@0"
I20260927 04:02:12.531843 697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/ts-0/wal/instance:
uuid: "f6c26349bd834fe8b3f4714e2f2203ea"
format_stamp: "Formatted at 2026-09-27 04:02:12 on dist-test-slave-glgz"
server_key: "158292456155e792693e957f272f133e"
server_key_iv: "89b6d2c1135e19d4af44854e5942bcdb"
server_key_version: "encryptionkey@0"
I20260927 04:02:12.533099 697 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 04:02:12.533818 712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:12.534013 697 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:02:12.534101 697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/ts-0/wal
uuid: "f6c26349bd834fe8b3f4714e2f2203ea"
format_stamp: "Formatted at 2026-09-27 04:02:12 on dist-test-slave-glgz"
server_key: "158292456155e792693e957f272f133e"
server_key_iv: "89b6d2c1135e19d4af44854e5942bcdb"
server_key_version: "encryptionkey@0"
I20260927 04:02:12.534176 697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:02:12.552310 697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:02:12.554358 697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:02:12.554507 697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:02:12.554797 697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:02:12.555147 697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:02:12.555207 697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:12.555248 697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:02:12.555276 697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:12.561190 697 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.65:46101
I20260927 04:02:12.561306 826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.65:46101 every 8 connection(s)
I20260927 04:02:12.561583 697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
Sep 27 04:02:12 dist-test-slave-glgz krb5kdc[589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481732, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.65@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 04:02:12.570510 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 697
I20260927 04:02:12.570699 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/ts-0/wal/instance
I20260927 04:02:12.570919 16101 external_mini_cluster.cc:1428] Setting key 3fa8b86f4b7fcdb84314bf550d053914
I20260927 04:02:12.572108 674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:12.562415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:43293 (local address 127.15.185.126:36371)
0927 04:02:12.562502 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:12.562504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:12.562704 (+ 200us) server_negotiation.cc:407] Connection header received
0927 04:02:12.563391 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:12.563393 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:12.563406 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:12.563457 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:12.564522 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:12.565093 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:12.566226 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:12.566478 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:12.566582 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:12.569045 (+ 2463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:12.569047 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:12.569048 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:12.569056 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:12.570463 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:12.571003 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:12.571005 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:12.571005 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:12.571097 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:12.571342 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:12.571343 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:12.571343 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:12.571487 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:12.571554 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:12.571969 (+ 415us) server_negotiation.cc:299] Negotiation successful
0927 04:02:12.572043 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 04:02:12.572429 829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:12.562389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:36371 (local address 127.15.185.65:43293)
0927 04:02:12.562563 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:12.562613 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:12.563268 (+ 655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:12.563533 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:12.563538 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:12.564184 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:02:12.564399 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:12.564404 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:12.565205 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:12.565206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:12.566111 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:12.566113 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:12.566595 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:12.566596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:12.566739 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:12.567423 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:02:12.567432 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:02:12.568937 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:02:12.570587 (+ 1650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:12.570595 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:12.570597 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:12.570907 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:12.571174 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:12.571175 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:12.571175 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:12.571287 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:12.571562 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:02:12.571569 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:02:12.571810 (+ 241us) client_negotiation.cc:769] Sending connection context
0927 04:02:12.571945 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 04:02:12.572137 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":80,"threads_started":1}
I20260927 04:02:12.572971 827 heartbeater.cc:347] Connected to a master server at 127.15.185.126:36371
I20260927 04:02:12.573096 827 heartbeater.cc:464] Registering TS with master...
I20260927 04:02:12.573292 827 heartbeater.cc:511] Master 127.15.185.126:36371 requested a full tablet report, sending...
I20260927 04:02:12.573911 634 ts_manager.cc:194] Registered new tserver with Master: f6c26349bd834fe8b3f4714e2f2203ea (127.15.185.65:46101)
I20260927 04:02:12.574779 634 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.185.65@KRBTEST.COM'} at 127.15.185.65:43293
WARNING: no policy specified for kudu/127.15.185.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab.
Entry for principal kudu/127.15.185.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab.
I20260927 04:02:12.598989 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.66:0
--local_ip_for_outbound_sockets=127.15.185.66
--webserver_interface=127.15.185.66
--webserver_port=0
--tserver_master_addrs=127.15.185.126:36371
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab
--principal=kudu/127.15.185.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:46511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:02:12.682622 832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:02:12.682825 832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:02:12.682849 832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:02:12.684093 832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:02:12.684136 832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:02:12.684163 832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.66
I20260927 04:02:12.685607 832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:46511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab
--principal=kudu/127.15.185.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.185.66
--webserver_port=0
--tserver_master_addrs=127.15.185.126:36371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.66
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:02:12.685868 832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:02:12.686108 832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 04:02:12.691476 832 server_base.cc:1034] running on GCE node
W20260927 04:02:12.691421 837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:12.691428 838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:12.691428 840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:12.691897 832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:02:12.692108 832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:02:12.693270 832 hybrid_clock.cc:648] HybridClock initialized: now 1790481732693257 us; error 28 us; skew 500 ppm
Sep 27 04:02:12 dist-test-slave-glgz krb5kdc[589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481732, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:02:12.695886 832 init.cc:377] Logged in from keytab as kudu/127.15.185.66@KRBTEST.COM (short username kudu)
I20260927 04:02:12.696337 832 webserver.cc:460] Webserver started at http://127.15.185.66:32975/ using document root <none> and password file <none>
I20260927 04:02:12.696636 832 fs_manager.cc:359] Metadata directory not provided
I20260927 04:02:12.696687 832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:02:12.696866 832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:02:12.697933 832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/ts-1/data/instance:
uuid: "563e85523f2f4343a29144cf6c4b46dc"
format_stamp: "Formatted at 2026-09-27 04:02:12 on dist-test-slave-glgz"
server_key: "0c383f0cda1113aac32ea6556c88d60c"
server_key_iv: "eb8ca59d5801f6520adc9147856c1034"
server_key_version: "encryptionkey@0"
I20260927 04:02:12.698292 832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/ts-1/wal/instance:
uuid: "563e85523f2f4343a29144cf6c4b46dc"
format_stamp: "Formatted at 2026-09-27 04:02:12 on dist-test-slave-glgz"
server_key: "0c383f0cda1113aac32ea6556c88d60c"
server_key_iv: "eb8ca59d5801f6520adc9147856c1034"
server_key_version: "encryptionkey@0"
I20260927 04:02:12.699813 832 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 04:02:12.700642 847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:12.700875 832 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:02:12.700978 832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/ts-1/wal
uuid: "563e85523f2f4343a29144cf6c4b46dc"
format_stamp: "Formatted at 2026-09-27 04:02:12 on dist-test-slave-glgz"
server_key: "0c383f0cda1113aac32ea6556c88d60c"
server_key_iv: "eb8ca59d5801f6520adc9147856c1034"
server_key_version: "encryptionkey@0"
I20260927 04:02:12.701054 832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:02:12.709024 832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:02:12.710971 832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:02:12.711159 832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:02:12.711374 832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:02:12.711673 832 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:02:12.711727 832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:12.711767 832 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:02:12.711796 832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:12.717633 832 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.66:42741
I20260927 04:02:12.717674 961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.66:42741 every 8 connection(s)
I20260927 04:02:12.718026 832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
I20260927 04:02:12.724191 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 832
I20260927 04:02:12.724334 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/ts-1/wal/instance
I20260927 04:02:12.724560 16101 external_mini_cluster.cc:1428] Setting key 26121526f03b3980e9048c7f46a2fc26
Sep 27 04:02:12 dist-test-slave-glgz krb5kdc[589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481732, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.66@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 04:02:12.728072 674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:12.718769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:37977 (local address 127.15.185.126:36371)
0927 04:02:12.718854 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:12.718856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:12.719088 (+ 232us) server_negotiation.cc:407] Connection header received
0927 04:02:12.719701 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:12.719703 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:12.719717 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:12.719790 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:12.720885 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:12.721386 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:12.722646 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:12.722845 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:12.722930 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:12.725271 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:12.725272 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:12.725272 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:12.725280 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:12.726731 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:12.727145 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:12.727146 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:12.727147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:12.727214 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:12.727459 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:12.727460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:12.727460 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:12.727617 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:12.727695 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:12.727936 (+ 241us) server_negotiation.cc:299] Negotiation successful
0927 04:02:12.728002 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 04:02:12.728237 964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:12.718772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:36371 (local address 127.15.185.66:37977)
0927 04:02:12.718939 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:12.718980 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:12.719550 (+ 570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:12.719867 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:12.719872 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:12.720591 (+ 719us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:02:12.720786 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:12.720791 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:12.721479 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:12.721480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:12.722543 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:12.722545 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:12.722940 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:12.722941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:12.723122 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:12.723590 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:02:12.723598 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:02:12.725159 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:02:12.726820 (+ 1661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:12.726826 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:12.726828 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:12.727081 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:12.727283 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:12.727283 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:12.727284 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:12.727385 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:12.727690 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:02:12.727694 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:02:12.727810 (+ 116us) client_negotiation.cc:769] Sending connection context
0927 04:02:12.727887 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 04:02:12.728010 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":70,"threads_started":1}
I20260927 04:02:12.728730 962 heartbeater.cc:347] Connected to a master server at 127.15.185.126:36371
I20260927 04:02:12.728860 962 heartbeater.cc:464] Registering TS with master...
I20260927 04:02:12.729041 962 heartbeater.cc:511] Master 127.15.185.126:36371 requested a full tablet report, sending...
I20260927 04:02:12.729486 634 ts_manager.cc:194] Registered new tserver with Master: 563e85523f2f4343a29144cf6c4b46dc (127.15.185.66:42741)
I20260927 04:02:12.729921 634 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.185.66@KRBTEST.COM'} at 127.15.185.66:37977
WARNING: no policy specified for kudu/127.15.185.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab.
Entry for principal kudu/127.15.185.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab.
I20260927 04:02:12.752228 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.67:0
--local_ip_for_outbound_sockets=127.15.185.67
--webserver_interface=127.15.185.67
--webserver_port=0
--tserver_master_addrs=127.15.185.126:36371
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab
--principal=kudu/127.15.185.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:46511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:02:12.837080 967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:02:12.837318 967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:02:12.837343 967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:02:12.838478 967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:02:12.838521 967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:02:12.838547 967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.67
I20260927 04:02:12.839942 967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:46511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab
--principal=kudu/127.15.185.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.185.67
--webserver_port=0
--tserver_master_addrs=127.15.185.126:36371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.67
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:02:12.840130 967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:02:12.840307 967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:02:12.845410 973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:12.845522 967 server_base.cc:1034] running on GCE node
W20260927 04:02:12.845547 975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:12.845422 972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:12.845798 967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:02:12.846048 967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:02:12.847239 967 hybrid_clock.cc:648] HybridClock initialized: now 1790481732847226 us; error 39 us; skew 500 ppm
Sep 27 04:02:12 dist-test-slave-glgz krb5kdc[589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481732, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:02:12.850080 967 init.cc:377] Logged in from keytab as kudu/127.15.185.67@KRBTEST.COM (short username kudu)
I20260927 04:02:12.850517 967 webserver.cc:460] Webserver started at http://127.15.185.67:34227/ using document root <none> and password file <none>
I20260927 04:02:12.850750 967 fs_manager.cc:359] Metadata directory not provided
I20260927 04:02:12.850796 967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:02:12.850934 967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:02:12.852023 967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/ts-2/data/instance:
uuid: "868aecde2f884e32bb8dc71b99632ac2"
format_stamp: "Formatted at 2026-09-27 04:02:12 on dist-test-slave-glgz"
server_key: "4928153b911975dd24ba1f353cc85ebd"
server_key_iv: "c9f701854e7adc686da0cad491a25b3e"
server_key_version: "encryptionkey@0"
I20260927 04:02:12.852401 967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/ts-2/wal/instance:
uuid: "868aecde2f884e32bb8dc71b99632ac2"
format_stamp: "Formatted at 2026-09-27 04:02:12 on dist-test-slave-glgz"
server_key: "4928153b911975dd24ba1f353cc85ebd"
server_key_iv: "c9f701854e7adc686da0cad491a25b3e"
server_key_version: "encryptionkey@0"
I20260927 04:02:12.853722 967 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 04:02:12.854483 982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:12.854719 967 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:02:12.854815 967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/ts-2/wal
uuid: "868aecde2f884e32bb8dc71b99632ac2"
format_stamp: "Formatted at 2026-09-27 04:02:12 on dist-test-slave-glgz"
server_key: "4928153b911975dd24ba1f353cc85ebd"
server_key_iv: "c9f701854e7adc686da0cad491a25b3e"
server_key_version: "encryptionkey@0"
I20260927 04:02:12.854867 967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:02:12.886461 967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:02:12.888842 967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:02:12.889021 967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:02:12.889278 967 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:02:12.889606 967 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:02:12.889659 967 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:12.889695 967 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:02:12.889724 967 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:12.895607 967 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.67:45273
I20260927 04:02:12.895694 1096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.67:45273 every 8 connection(s)
I20260927 04:02:12.896016 967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
I20260927 04:02:12.897143 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 967
I20260927 04:02:12.897267 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/ts-2/wal/instance
I20260927 04:02:12.897444 16101 external_mini_cluster.cc:1428] Setting key 63023f11bb335ff70e90351f16e27497
Sep 27 04:02:12 dist-test-slave-glgz krb5kdc[589](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481732, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.67@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 04:02:12.905601 674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:12.896867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:47729 (local address 127.15.185.126:36371)
0927 04:02:12.896912 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:12.896915 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:12.897041 (+ 126us) server_negotiation.cc:407] Connection header received
0927 04:02:12.897628 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:12.897630 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:12.897642 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:12.897746 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:12.898870 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:12.899465 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:12.900439 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:12.900622 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:12.900673 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:12.902844 (+ 2171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:12.902846 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:12.902846 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:12.902854 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:12.904358 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:12.904769 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:12.904770 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:12.904770 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:12.904857 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:12.905075 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:12.905076 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:12.905076 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:12.905180 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:12.905242 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:12.905470 (+ 228us) server_negotiation.cc:299] Negotiation successful
0927 04:02:12.905533 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 04:02:12.905833 1099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:12.896717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:36371 (local address 127.15.185.67:47729)
0927 04:02:12.896885 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:12.896934 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:12.897491 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:12.897836 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:12.897840 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:12.898571 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:02:12.898784 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:12.898789 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:12.899545 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:12.899546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:12.900361 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:12.900364 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:12.900706 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:12.900707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:12.900810 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:12.901285 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:02:12.901294 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:02:12.902742 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:02:12.904432 (+ 1690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:12.904437 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:12.904439 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:12.904659 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:12.904916 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:12.904917 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:12.904917 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:12.905020 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:12.905235 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:02:12.905239 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:02:12.905356 (+ 117us) client_negotiation.cc:769] Sending connection context
0927 04:02:12.905441 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 04:02:12.905563 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":98,"threads_started":1}
I20260927 04:02:12.906373 1097 heartbeater.cc:347] Connected to a master server at 127.15.185.126:36371
I20260927 04:02:12.906478 1097 heartbeater.cc:464] Registering TS with master...
I20260927 04:02:12.906680 1097 heartbeater.cc:511] Master 127.15.185.126:36371 requested a full tablet report, sending...
I20260927 04:02:12.907127 634 ts_manager.cc:194] Registered new tserver with Master: 868aecde2f884e32bb8dc71b99632ac2 (127.15.185.67:45273)
I20260927 04:02:12.907507 634 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.185.67@KRBTEST.COM'} at 127.15.185.67:47729
I20260927 04:02:12.908988 16101 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 04:02:12.916234 674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:12.910253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56560 (local address 127.15.185.126:36371)
0927 04:02:12.910316 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:12.910318 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:12.910442 (+ 124us) server_negotiation.cc:407] Connection header received
0927 04:02:12.910493 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:12.910494 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:12.910503 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:12.910554 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:12.911110 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:12.911569 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:12.912314 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:12.912502 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:12.912571 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:12.913694 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:12.913695 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:12.913696 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:12.913702 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:12.915244 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:12.915561 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:12.915563 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:12.915563 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:12.915631 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:12.915811 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:12.915811 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:12.915811 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:12.915963 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:12.916037 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:12.916108 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 04:02:12.916167 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 04:02:12.917757 634 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56560:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 04:02:12.919435 634 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 04:02:12.926147 1113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:12.921990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55978 (local address 127.15.185.66:42741)
0927 04:02:12.922194 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:12.922198 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:12.922582 (+ 384us) server_negotiation.cc:407] Connection header received
0927 04:02:12.922626 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:12.922628 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:12.922645 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:12.922713 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:02:12.922925 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:12.923682 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:12.925222 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:12.925705 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:12.925794 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:12.925857 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:12.926001 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 04:02:12.926045 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":86,"threads_started":1}
I20260927 04:02:12.926188 1110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:12.921733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.66:42741 (local address 127.0.0.1:55978)
0927 04:02:12.922460 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:12.922469 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:12.922520 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:12.922799 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:12.922800 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:12.922801 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:02:12.922850 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:12.922851 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:12.923782 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:12.923783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:12.925120 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:12.925122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:12.925793 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:12.925794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:12.925880 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:12.925893 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 04:02:12.925985 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 04:02:12.926045 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"mutex_wait_us":5,"thread_start_us":65,"threads_started":1}
I20260927 04:02:12.926434 1112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:12.921952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33334 (local address 127.15.185.67:45273)
0927 04:02:12.922159 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:12.922164 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:12.922173 (+ 9us) server_negotiation.cc:407] Connection header received
0927 04:02:12.922223 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:12.922224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:12.922238 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:12.922348 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:02:12.922536 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:12.923723 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:12.925018 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:12.925730 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:12.925803 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:12.925857 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:12.926275 (+ 418us) server_negotiation.cc:299] Negotiation successful
0927 04:02:12.926335 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":81,"threads_started":1}
I20260927 04:02:12.926519 1109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:12.921671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.65:46101 (local address 127.0.0.1:51886)
0927 04:02:12.921954 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:12.922001 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:12.922066 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:12.922531 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:12.922532 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:12.922534 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:02:12.922567 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:12.922569 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:12.923872 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:12.923872 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:12.925120 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:12.925122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:12.926286 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:12.926287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:12.926353 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:12.926357 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 04:02:12.926411 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 04:02:12.926451 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":89,"thread_start_us":76,"threads_started":1}
I20260927 04:02:12.926908 1111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:12.921778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51886 (local address 127.15.185.65:46101)
0927 04:02:12.922281 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:12.922284 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:12.922293 (+ 9us) server_negotiation.cc:407] Connection header received
0927 04:02:12.922348 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:12.922350 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:12.922366 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:12.922446 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:02:12.922644 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:12.923782 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:12.925219 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:12.925913 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:12.925999 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:12.926049 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:12.926756 (+ 707us) server_negotiation.cc:299] Negotiation successful
0927 04:02:12.926818 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":82,"threads_started":1}
I20260927 04:02:12.926174 1108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:12.921703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.67:45273 (local address 127.0.0.1:33334)
0927 04:02:12.921956 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:12.922000 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:12.922068 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:12.922366 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:12.922370 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:12.922372 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:02:12.922449 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:12.922451 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:12.923841 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:12.923842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:12.924922 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:12.924925 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:12.925894 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:12.925895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:12.925961 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:12.925966 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 04:02:12.926013 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 04:02:12.926061 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":74,"thread_start_us":95,"threads_started":1}
I20260927 04:02:12.926980 895 tablet_service.cc:1467] Processing CreateTablet for tablet 2bda06f88abf4ad4bd46d03289944c31 (DEFAULT_TABLE table=test-table [id=5ec4b970dc5b410b9148ab1b84abff90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:02:12.927415 895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bda06f88abf4ad4bd46d03289944c31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:02:12.927623 760 tablet_service.cc:1467] Processing CreateTablet for tablet 2bda06f88abf4ad4bd46d03289944c31 (DEFAULT_TABLE table=test-table [id=5ec4b970dc5b410b9148ab1b84abff90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:02:12.927875 760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bda06f88abf4ad4bd46d03289944c31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:02:12.927780 1030 tablet_service.cc:1467] Processing CreateTablet for tablet 2bda06f88abf4ad4bd46d03289944c31 (DEFAULT_TABLE table=test-table [id=5ec4b970dc5b410b9148ab1b84abff90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:02:12.928102 1030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bda06f88abf4ad4bd46d03289944c31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:02:12.930550 1114 tablet_bootstrap.cc:492] T 2bda06f88abf4ad4bd46d03289944c31 P f6c26349bd834fe8b3f4714e2f2203ea: Bootstrap starting.
I20260927 04:02:12.931133 1115 tablet_bootstrap.cc:492] T 2bda06f88abf4ad4bd46d03289944c31 P 563e85523f2f4343a29144cf6c4b46dc: Bootstrap starting.
I20260927 04:02:12.931293 1114 tablet_bootstrap.cc:654] T 2bda06f88abf4ad4bd46d03289944c31 P f6c26349bd834fe8b3f4714e2f2203ea: Neither blocks nor log segments found. Creating new log.
I20260927 04:02:12.931607 1116 tablet_bootstrap.cc:492] T 2bda06f88abf4ad4bd46d03289944c31 P 868aecde2f884e32bb8dc71b99632ac2: Bootstrap starting.
I20260927 04:02:12.931840 1115 tablet_bootstrap.cc:654] T 2bda06f88abf4ad4bd46d03289944c31 P 563e85523f2f4343a29144cf6c4b46dc: Neither blocks nor log segments found. Creating new log.
I20260927 04:02:12.931960 1114 log.cc:824] T 2bda06f88abf4ad4bd46d03289944c31 P f6c26349bd834fe8b3f4714e2f2203ea: Log is configured to *not* fsync() on all Append() calls
I20260927 04:02:12.932258 1116 tablet_bootstrap.cc:654] T 2bda06f88abf4ad4bd46d03289944c31 P 868aecde2f884e32bb8dc71b99632ac2: Neither blocks nor log segments found. Creating new log.
I20260927 04:02:12.932390 1115 log.cc:824] T 2bda06f88abf4ad4bd46d03289944c31 P 563e85523f2f4343a29144cf6c4b46dc: Log is configured to *not* fsync() on all Append() calls
I20260927 04:02:12.932857 1114 tablet_bootstrap.cc:492] T 2bda06f88abf4ad4bd46d03289944c31 P f6c26349bd834fe8b3f4714e2f2203ea: No bootstrap required, opened a new log
I20260927 04:02:12.932843 1116 log.cc:824] T 2bda06f88abf4ad4bd46d03289944c31 P 868aecde2f884e32bb8dc71b99632ac2: Log is configured to *not* fsync() on all Append() calls
I20260927 04:02:12.932930 1114 ts_tablet_manager.cc:1397] T 2bda06f88abf4ad4bd46d03289944c31 P f6c26349bd834fe8b3f4714e2f2203ea: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 04:02:12.933135 1115 tablet_bootstrap.cc:492] T 2bda06f88abf4ad4bd46d03289944c31 P 563e85523f2f4343a29144cf6c4b46dc: No bootstrap required, opened a new log
I20260927 04:02:12.933256 1115 ts_tablet_manager.cc:1397] T 2bda06f88abf4ad4bd46d03289944c31 P 563e85523f2f4343a29144cf6c4b46dc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 04:02:12.933565 1116 tablet_bootstrap.cc:492] T 2bda06f88abf4ad4bd46d03289944c31 P 868aecde2f884e32bb8dc71b99632ac2: No bootstrap required, opened a new log
I20260927 04:02:12.933629 1116 ts_tablet_manager.cc:1397] T 2bda06f88abf4ad4bd46d03289944c31 P 868aecde2f884e32bb8dc71b99632ac2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 04:02:12.934682 1115 raft_consensus.cc:348] T 2bda06f88abf4ad4bd46d03289944c31 P 563e85523f2f4343a29144cf6c4b46dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6c26349bd834fe8b3f4714e2f2203ea" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46101 } } peers { permanent_uuid: "868aecde2f884e32bb8dc71b99632ac2" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 45273 } } peers { permanent_uuid: "563e85523f2f4343a29144cf6c4b46dc" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 42741 } }
I20260927 04:02:12.934824 1115 raft_consensus.cc:374] T 2bda06f88abf4ad4bd46d03289944c31 P 563e85523f2f4343a29144cf6c4b46dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:02:12.934847 1115 raft_consensus.cc:729] T 2bda06f88abf4ad4bd46d03289944c31 P 563e85523f2f4343a29144cf6c4b46dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 563e85523f2f4343a29144cf6c4b46dc, State: Initialized, Role: FOLLOWER
I20260927 04:02:12.934973 1115 consensus_queue.cc:260] T 2bda06f88abf4ad4bd46d03289944c31 P 563e85523f2f4343a29144cf6c4b46dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6c26349bd834fe8b3f4714e2f2203ea" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46101 } } peers { permanent_uuid: "868aecde2f884e32bb8dc71b99632ac2" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 45273 } } peers { permanent_uuid: "563e85523f2f4343a29144cf6c4b46dc" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 42741 } }
I20260927 04:02:12.935108 1114 raft_consensus.cc:348] T 2bda06f88abf4ad4bd46d03289944c31 P f6c26349bd834fe8b3f4714e2f2203ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6c26349bd834fe8b3f4714e2f2203ea" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46101 } } peers { permanent_uuid: "868aecde2f884e32bb8dc71b99632ac2" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 45273 } } peers { permanent_uuid: "563e85523f2f4343a29144cf6c4b46dc" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 42741 } }
I20260927 04:02:12.935283 1114 raft_consensus.cc:374] T 2bda06f88abf4ad4bd46d03289944c31 P f6c26349bd834fe8b3f4714e2f2203ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:02:12.935324 1114 raft_consensus.cc:729] T 2bda06f88abf4ad4bd46d03289944c31 P f6c26349bd834fe8b3f4714e2f2203ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6c26349bd834fe8b3f4714e2f2203ea, State: Initialized, Role: FOLLOWER
I20260927 04:02:12.935456 1115 ts_tablet_manager.cc:1428] T 2bda06f88abf4ad4bd46d03289944c31 P 563e85523f2f4343a29144cf6c4b46dc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 04:02:12.935509 962 heartbeater.cc:503] Master 127.15.185.126:36371 was elected leader, sending a full tablet report...
I20260927 04:02:12.935619 1116 raft_consensus.cc:348] T 2bda06f88abf4ad4bd46d03289944c31 P 868aecde2f884e32bb8dc71b99632ac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6c26349bd834fe8b3f4714e2f2203ea" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46101 } } peers { permanent_uuid: "868aecde2f884e32bb8dc71b99632ac2" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 45273 } } peers { permanent_uuid: "563e85523f2f4343a29144cf6c4b46dc" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 42741 } }
I20260927 04:02:12.935783 1116 raft_consensus.cc:374] T 2bda06f88abf4ad4bd46d03289944c31 P 868aecde2f884e32bb8dc71b99632ac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:02:12.935823 1116 raft_consensus.cc:729] T 2bda06f88abf4ad4bd46d03289944c31 P 868aecde2f884e32bb8dc71b99632ac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 868aecde2f884e32bb8dc71b99632ac2, State: Initialized, Role: FOLLOWER
I20260927 04:02:12.935827 1114 consensus_queue.cc:260] T 2bda06f88abf4ad4bd46d03289944c31 P f6c26349bd834fe8b3f4714e2f2203ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6c26349bd834fe8b3f4714e2f2203ea" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46101 } } peers { permanent_uuid: "868aecde2f884e32bb8dc71b99632ac2" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 45273 } } peers { permanent_uuid: "563e85523f2f4343a29144cf6c4b46dc" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 42741 } }
I20260927 04:02:12.935931 1116 consensus_queue.cc:260] T 2bda06f88abf4ad4bd46d03289944c31 P 868aecde2f884e32bb8dc71b99632ac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6c26349bd834fe8b3f4714e2f2203ea" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46101 } } peers { permanent_uuid: "868aecde2f884e32bb8dc71b99632ac2" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 45273 } } peers { permanent_uuid: "563e85523f2f4343a29144cf6c4b46dc" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 42741 } }
I20260927 04:02:12.936161 1114 ts_tablet_manager.cc:1428] T 2bda06f88abf4ad4bd46d03289944c31 P f6c26349bd834fe8b3f4714e2f2203ea: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 04:02:12.936208 1116 ts_tablet_manager.cc:1428] T 2bda06f88abf4ad4bd46d03289944c31 P 868aecde2f884e32bb8dc71b99632ac2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 04:02:12.936280 827 heartbeater.cc:503] Master 127.15.185.126:36371 was elected leader, sending a full tablet report...
I20260927 04:02:12.936544 1097 heartbeater.cc:503] Master 127.15.185.126:36371 was elected leader, sending a full tablet report...
I20260927 04:02:12.937121 1122 raft_consensus.cc:482] T 2bda06f88abf4ad4bd46d03289944c31 P 868aecde2f884e32bb8dc71b99632ac2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:02:12.937211 1122 raft_consensus.cc:504] T 2bda06f88abf4ad4bd46d03289944c31 P 868aecde2f884e32bb8dc71b99632ac2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6c26349bd834fe8b3f4714e2f2203ea" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46101 } } peers { permanent_uuid: "868aecde2f884e32bb8dc71b99632ac2" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 45273 } } peers { permanent_uuid: "563e85523f2f4343a29144cf6c4b46dc" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 42741 } }
I20260927 04:02:12.937901 1122 leader_election.cc:288] T 2bda06f88abf4ad4bd46d03289944c31 P 868aecde2f884e32bb8dc71b99632ac2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6c26349bd834fe8b3f4714e2f2203ea (127.15.185.65:46101), 563e85523f2f4343a29144cf6c4b46dc (127.15.185.66:42741)
I20260927 04:02:12.940770 1111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:12.938062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:45103 (local address 127.15.185.65:46101)
0927 04:02:12.938108 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:12.938111 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:12.938128 (+ 17us) server_negotiation.cc:407] Connection header received
0927 04:02:12.938185 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:12.938186 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:12.938197 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:12.938244 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:02:12.938452 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:12.938972 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:12.939994 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:12.940397 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:12.940456 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:12.940463 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:12.940643 (+ 180us) server_negotiation.cc:299] Negotiation successful
0927 04:02:12.940693 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 04:02:12.940768 1099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:12.938025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.65:46101 (local address 127.15.185.67:45103)
0927 04:02:12.938063 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:12.938075 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:12.938119 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:12.938295 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:12.938296 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:12.938298 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:02:12.938371 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:12.938372 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:12.939090 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:12.939091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:12.939901 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:12.939903 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:12.940480 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:12.940481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:12.940567 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:12.940570 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 04:02:12.940624 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 04:02:12.940680 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19,"mutex_wait_us":46}
I20260927 04:02:12.940827 1113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:12.938244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:53757 (local address 127.15.185.66:42741)
0927 04:02:12.938285 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:12.938287 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:12.938294 (+ 7us) server_negotiation.cc:407] Connection header received
0927 04:02:12.938314 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:12.938315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:12.938323 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:12.938384 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:02:12.938540 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:12.938976 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:12.940038 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:12.940412 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:12.940509 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:12.940517 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:12.940703 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 04:02:12.940746 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 04:02:12.940902 1123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:12.938046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.66:42741 (local address 127.15.185.67:53757)
0927 04:02:12.938184 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:12.938196 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:12.938255 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:12.938386 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:12.938387 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:12.938388 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:02:12.938477 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:12.938479 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:12.939079 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:12.939080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:12.939919 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:12.939921 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:12.940524 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:12.940525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:12.940619 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:12.940624 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 04:02:12.940682 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 04:02:12.940721 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":29,"thread_start_us":93,"threads_started":1}
I20260927 04:02:12.941080 780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bda06f88abf4ad4bd46d03289944c31" candidate_uuid: "868aecde2f884e32bb8dc71b99632ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6c26349bd834fe8b3f4714e2f2203ea" is_pre_election: true
I20260927 04:02:12.941268 780 raft_consensus.cc:2454] T 2bda06f88abf4ad4bd46d03289944c31 P f6c26349bd834fe8b3f4714e2f2203ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 868aecde2f884e32bb8dc71b99632ac2 in term 0.
I20260927 04:02:12.941233 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bda06f88abf4ad4bd46d03289944c31" candidate_uuid: "868aecde2f884e32bb8dc71b99632ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "563e85523f2f4343a29144cf6c4b46dc" is_pre_election: true
I20260927 04:02:12.941356 916 raft_consensus.cc:2454] T 2bda06f88abf4ad4bd46d03289944c31 P 563e85523f2f4343a29144cf6c4b46dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 868aecde2f884e32bb8dc71b99632ac2 in term 0.
I20260927 04:02:12.941502 984 leader_election.cc:302] T 2bda06f88abf4ad4bd46d03289944c31 P 868aecde2f884e32bb8dc71b99632ac2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 868aecde2f884e32bb8dc71b99632ac2, f6c26349bd834fe8b3f4714e2f2203ea; no voters:
I20260927 04:02:12.941627 1122 raft_consensus.cc:2781] T 2bda06f88abf4ad4bd46d03289944c31 P 868aecde2f884e32bb8dc71b99632ac2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 04:02:12.941680 1122 raft_consensus.cc:482] T 2bda06f88abf4ad4bd46d03289944c31 P 868aecde2f884e32bb8dc71b99632ac2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 04:02:12.941704 1122 raft_consensus.cc:3037] T 2bda06f88abf4ad4bd46d03289944c31 P 868aecde2f884e32bb8dc71b99632ac2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:02:12.942426 1122 raft_consensus.cc:504] T 2bda06f88abf4ad4bd46d03289944c31 P 868aecde2f884e32bb8dc71b99632ac2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6c26349bd834fe8b3f4714e2f2203ea" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46101 } } peers { permanent_uuid: "868aecde2f884e32bb8dc71b99632ac2" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 45273 } } peers { permanent_uuid: "563e85523f2f4343a29144cf6c4b46dc" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 42741 } }
I20260927 04:02:12.942597 1122 leader_election.cc:288] T 2bda06f88abf4ad4bd46d03289944c31 P 868aecde2f884e32bb8dc71b99632ac2 [CANDIDATE]: Term 1 election: Requested vote from peers f6c26349bd834fe8b3f4714e2f2203ea (127.15.185.65:46101), 563e85523f2f4343a29144cf6c4b46dc (127.15.185.66:42741)
I20260927 04:02:12.942749 780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bda06f88abf4ad4bd46d03289944c31" candidate_uuid: "868aecde2f884e32bb8dc71b99632ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6c26349bd834fe8b3f4714e2f2203ea"
I20260927 04:02:12.942762 916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bda06f88abf4ad4bd46d03289944c31" candidate_uuid: "868aecde2f884e32bb8dc71b99632ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "563e85523f2f4343a29144cf6c4b46dc"
I20260927 04:02:12.942830 916 raft_consensus.cc:3037] T 2bda06f88abf4ad4bd46d03289944c31 P 563e85523f2f4343a29144cf6c4b46dc [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:02:12.942830 780 raft_consensus.cc:3037] T 2bda06f88abf4ad4bd46d03289944c31 P f6c26349bd834fe8b3f4714e2f2203ea [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:02:12.943783 780 raft_consensus.cc:2454] T 2bda06f88abf4ad4bd46d03289944c31 P f6c26349bd834fe8b3f4714e2f2203ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 868aecde2f884e32bb8dc71b99632ac2 in term 1.
I20260927 04:02:12.943783 916 raft_consensus.cc:2454] T 2bda06f88abf4ad4bd46d03289944c31 P 563e85523f2f4343a29144cf6c4b46dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 868aecde2f884e32bb8dc71b99632ac2 in term 1.
I20260927 04:02:12.943981 983 leader_election.cc:302] T 2bda06f88abf4ad4bd46d03289944c31 P 868aecde2f884e32bb8dc71b99632ac2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 563e85523f2f4343a29144cf6c4b46dc, 868aecde2f884e32bb8dc71b99632ac2; no voters:
I20260927 04:02:12.944083 1122 raft_consensus.cc:2781] T 2bda06f88abf4ad4bd46d03289944c31 P 868aecde2f884e32bb8dc71b99632ac2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:02:12.944262 1122 raft_consensus.cc:686] T 2bda06f88abf4ad4bd46d03289944c31 P 868aecde2f884e32bb8dc71b99632ac2 [term 1 LEADER]: Becoming Leader. State: Replica: 868aecde2f884e32bb8dc71b99632ac2, State: Running, Role: LEADER
I20260927 04:02:12.944382 1122 consensus_queue.cc:237] T 2bda06f88abf4ad4bd46d03289944c31 P 868aecde2f884e32bb8dc71b99632ac2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6c26349bd834fe8b3f4714e2f2203ea" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46101 } } peers { permanent_uuid: "868aecde2f884e32bb8dc71b99632ac2" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 45273 } } peers { permanent_uuid: "563e85523f2f4343a29144cf6c4b46dc" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 42741 } }
I20260927 04:02:12.945327 632 catalog_manager.cc:5495] T 2bda06f88abf4ad4bd46d03289944c31 P 868aecde2f884e32bb8dc71b99632ac2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 868aecde2f884e32bb8dc71b99632ac2 (127.15.185.67). New cstate: current_term: 1 leader_uuid: "868aecde2f884e32bb8dc71b99632ac2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6c26349bd834fe8b3f4714e2f2203ea" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "868aecde2f884e32bb8dc71b99632ac2" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 45273 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "563e85523f2f4343a29144cf6c4b46dc" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 42741 } health_report { overall_health: UNKNOWN } } }
I20260927 04:02:12.955149 1112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:12.952829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33346 (local address 127.15.185.67:45273)
0927 04:02:12.952897 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:12.952902 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:12.952911 (+ 9us) server_negotiation.cc:407] Connection header received
0927 04:02:12.952945 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:12.952946 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:12.952963 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:12.953029 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:02:12.953122 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:12.953597 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:12.954335 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:12.954516 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:12.954576 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:12.954777 (+ 201us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:02:12.954876 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:02:12.954939 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:12.954973 (+ 34us) server_negotiation.cc:299] Negotiation successful
0927 04:02:12.954991 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 04:02:12.957063 916 raft_consensus.cc:1261] T 2bda06f88abf4ad4bd46d03289944c31 P 563e85523f2f4343a29144cf6c4b46dc [term 1 FOLLOWER]: Refusing update from remote peer 868aecde2f884e32bb8dc71b99632ac2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:02:12.957099 780 raft_consensus.cc:1261] T 2bda06f88abf4ad4bd46d03289944c31 P f6c26349bd834fe8b3f4714e2f2203ea [term 1 FOLLOWER]: Refusing update from remote peer 868aecde2f884e32bb8dc71b99632ac2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:02:12.957459 1124 consensus_queue.cc:1035] T 2bda06f88abf4ad4bd46d03289944c31 P 868aecde2f884e32bb8dc71b99632ac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "563e85523f2f4343a29144cf6c4b46dc" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 42741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:02:12.957602 1122 consensus_queue.cc:1035] T 2bda06f88abf4ad4bd46d03289944c31 P 868aecde2f884e32bb8dc71b99632ac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6c26349bd834fe8b3f4714e2f2203ea" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 46101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:02:12.960088 1127 mvcc.cc:205] Tried to move back new op lower bound from 7333813178189398016 to 7333813178141196288. Current Snapshot: MvccSnapshot[applied={T|T < 7333813178189398016}]
I20260927 04:02:12.960266 1129 mvcc.cc:205] Tried to move back new op lower bound from 7333813178189398016 to 7333813178141196288. Current Snapshot: MvccSnapshot[applied={T|T < 7333813178189398016}]
I20260927 04:02:12.962672 634 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:02:12.962785 634 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:02:12.963577 634 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bb208bd43f804dc085b4d803a6755f74: Sending DeleteTablet for 3 replicas of tablet 2bda06f88abf4ad4bd46d03289944c31
I20260927 04:02:12.963944 1030 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bda06f88abf4ad4bd46d03289944c31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:02:12 UTC) from {username='kudu', principal='kudu/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:33334
I20260927 04:02:12.963953 760 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bda06f88abf4ad4bd46d03289944c31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:02:12 UTC) from {username='kudu', principal='kudu/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:51886
I20260927 04:02:12.964254 895 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bda06f88abf4ad4bd46d03289944c31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:02:12 UTC) from {username='kudu', principal='kudu/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:55978
I20260927 04:02:12.964462 1139 tablet_replica.cc:330] T 2bda06f88abf4ad4bd46d03289944c31 P 563e85523f2f4343a29144cf6c4b46dc: stopping tablet replica
I20260927 04:02:12.964545 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 697
I20260927 04:02:12.964568 1139 raft_consensus.cc:2229] T 2bda06f88abf4ad4bd46d03289944c31 P 563e85523f2f4343a29144cf6c4b46dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:02:12.964672 1139 raft_consensus.cc:2258] T 2bda06f88abf4ad4bd46d03289944c31 P 563e85523f2f4343a29144cf6c4b46dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:02:12.966940 1139 ts_tablet_manager.cc:1905] T 2bda06f88abf4ad4bd46d03289944c31 P 563e85523f2f4343a29144cf6c4b46dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:02:12.969038 1139 ts_tablet_manager.cc:1918] T 2bda06f88abf4ad4bd46d03289944c31 P 563e85523f2f4343a29144cf6c4b46dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:02:12.969127 1139 log.cc:1196] T 2bda06f88abf4ad4bd46d03289944c31 P 563e85523f2f4343a29144cf6c4b46dc: Deleting WAL directory at /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/ts-1/wal/wals/2bda06f88abf4ad4bd46d03289944c31
I20260927 04:02:12.969419 1139 ts_tablet_manager.cc:1939] T 2bda06f88abf4ad4bd46d03289944c31 P 563e85523f2f4343a29144cf6c4b46dc: Deleting consensus metadata
I20260927 04:02:12.970117 618 catalog_manager.cc:4841] TS 563e85523f2f4343a29144cf6c4b46dc (127.15.185.66:42741): tablet 2bda06f88abf4ad4bd46d03289944c31 (table test-table [id=5ec4b970dc5b410b9148ab1b84abff90]) successfully deleted
W20260927 04:02:12.971094 619 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.185.65:46101 (error 108)
I20260927 04:02:12.971153 1138 tablet_replica.cc:330] T 2bda06f88abf4ad4bd46d03289944c31 P 868aecde2f884e32bb8dc71b99632ac2: stopping tablet replica
I20260927 04:02:12.971309 1138 raft_consensus.cc:2229] T 2bda06f88abf4ad4bd46d03289944c31 P 868aecde2f884e32bb8dc71b99632ac2 [term 1 LEADER]: Raft consensus shutting down.
I20260927 04:02:12.971431 1138 raft_consensus.cc:2258] T 2bda06f88abf4ad4bd46d03289944c31 P 868aecde2f884e32bb8dc71b99632ac2 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 04:02:12.971748 1110 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:02:12.971583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.65:46101 (local address 127.0.0.1:51898)
0927 04:02:12.971650 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:12.971705 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.65:46101: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 04:02:12.971891 619 catalog_manager.cc:4584] TS f6c26349bd834fe8b3f4714e2f2203ea (127.15.185.65:46101): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2bda06f88abf4ad4bd46d03289944c31: Network error: Client connection negotiation failed: client connection to 127.15.185.65:46101: connect: Connection reset by peer (error 104)
I20260927 04:02:12.971995 1138 ts_tablet_manager.cc:1905] T 2bda06f88abf4ad4bd46d03289944c31 P 868aecde2f884e32bb8dc71b99632ac2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:02:12.973376 1138 ts_tablet_manager.cc:1918] T 2bda06f88abf4ad4bd46d03289944c31 P 868aecde2f884e32bb8dc71b99632ac2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:02:12.973412 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 832
I20260927 04:02:12.973482 1138 log.cc:1196] T 2bda06f88abf4ad4bd46d03289944c31 P 868aecde2f884e32bb8dc71b99632ac2: Deleting WAL directory at /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481498018407-16101-0/minicluster-data/ts-2/wal/wals/2bda06f88abf4ad4bd46d03289944c31
I20260927 04:02:12.973775 1138 ts_tablet_manager.cc:1939] T 2bda06f88abf4ad4bd46d03289944c31 P 868aecde2f884e32bb8dc71b99632ac2: Deleting consensus metadata
I20260927 04:02:12.975131 620 catalog_manager.cc:4841] TS 868aecde2f884e32bb8dc71b99632ac2 (127.15.185.67:45273): tablet 2bda06f88abf4ad4bd46d03289944c31 (table test-table [id=5ec4b970dc5b410b9148ab1b84abff90]) successfully deleted
I20260927 04:02:12.980602 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 967
I20260927 04:02:12.985731 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 602
2026-09-27T04:02:12Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1925 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-27T04:02:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:02:13Z Disabled control of system clock
I20260927 04:02:13.013844 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:44297
--webserver_interface=127.15.185.126
--webserver_port=0
--builtin_ntp_servers=127.15.185.84:43239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.185.126:44297
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 04:02:13.096304 1144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:02:13.096544 1144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:02:13.096566 1144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:02:13.097838 1144 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:02:13.097893 1144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:02:13.097906 1144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:02:13.097958 1144 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:02:13.097981 1144 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:02:13.099449 1144 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:43239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.185.126:44297
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:44297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:02:13.099673 1144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:02:13.099931 1144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:02:13.105175 1149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:13.105191 1152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:13.105242 1150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:13.105516 1144 server_base.cc:1034] running on GCE node
I20260927 04:02:13.105938 1144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:02:13.106348 1144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:02:13.107530 1144 hybrid_clock.cc:648] HybridClock initialized: now 1790481733107511 us; error 43 us; skew 500 ppm
I20260927 04:02:13.109067 1144 webserver.cc:460] Webserver started at http://127.15.185.126:36085/ using document root <none> and password file <none>
I20260927 04:02:13.109378 1144 fs_manager.cc:359] Metadata directory not provided
I20260927 04:02:13.109455 1144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:02:13.109611 1144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:02:13.110716 1144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/master-0/data/instance:
uuid: "90dcafd31f39443386594cc5480318ec"
format_stamp: "Formatted at 2026-09-27 04:02:13 on dist-test-slave-glgz"
server_key: "3c36cf495e12cbda3b52d580d31b378c"
server_key_iv: "bd52c41ebc7d45372e44dbfb540af3d2"
server_key_version: "encryptionkey@0"
I20260927 04:02:13.111159 1144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/master-0/wal/instance:
uuid: "90dcafd31f39443386594cc5480318ec"
format_stamp: "Formatted at 2026-09-27 04:02:13 on dist-test-slave-glgz"
server_key: "3c36cf495e12cbda3b52d580d31b378c"
server_key_iv: "bd52c41ebc7d45372e44dbfb540af3d2"
server_key_version: "encryptionkey@0"
I20260927 04:02:13.112532 1144 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:02:13.113329 1158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:13.113503 1144 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:02:13.113616 1144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/master-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/master-0/wal
uuid: "90dcafd31f39443386594cc5480318ec"
format_stamp: "Formatted at 2026-09-27 04:02:13 on dist-test-slave-glgz"
server_key: "3c36cf495e12cbda3b52d580d31b378c"
server_key_iv: "bd52c41ebc7d45372e44dbfb540af3d2"
server_key_version: "encryptionkey@0"
I20260927 04:02:13.113705 1144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:02:13.140698 1144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:02:13.141081 1144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:02:13.141259 1144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:02:13.145467 1144 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.126:44297
I20260927 04:02:13.145511 1210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.126:44297 every 8 connection(s)
I20260927 04:02:13.145913 1144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
I20260927 04:02:13.146942 1211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:02:13.148906 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 1144
I20260927 04:02:13.149013 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/master-0/wal/instance
I20260927 04:02:13.149216 16101 external_mini_cluster.cc:1428] Setting key 161ce5637438e1f01178ffaaf9311da6
I20260927 04:02:13.149848 1211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90dcafd31f39443386594cc5480318ec: Bootstrap starting.
I20260927 04:02:13.150489 1211 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90dcafd31f39443386594cc5480318ec: Neither blocks nor log segments found. Creating new log.
I20260927 04:02:13.150861 1211 log.cc:824] T 00000000000000000000000000000000 P 90dcafd31f39443386594cc5480318ec: Log is configured to *not* fsync() on all Append() calls
I20260927 04:02:13.151609 1211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90dcafd31f39443386594cc5480318ec: No bootstrap required, opened a new log
I20260927 04:02:13.154064 1211 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90dcafd31f39443386594cc5480318ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90dcafd31f39443386594cc5480318ec" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 44297 } }
I20260927 04:02:13.154249 1211 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90dcafd31f39443386594cc5480318ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:02:13.154312 1211 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90dcafd31f39443386594cc5480318ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90dcafd31f39443386594cc5480318ec, State: Initialized, Role: FOLLOWER
I20260927 04:02:13.154477 1211 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90dcafd31f39443386594cc5480318ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90dcafd31f39443386594cc5480318ec" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 44297 } }
I20260927 04:02:13.154577 1211 raft_consensus.cc:388] T 00000000000000000000000000000000 P 90dcafd31f39443386594cc5480318ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:02:13.154623 1211 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90dcafd31f39443386594cc5480318ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:02:13.154666 1211 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90dcafd31f39443386594cc5480318ec [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:02:13.155339 1214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:13.149805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58696 (local address 127.15.185.126:44297)
0927 04:02:13.150032 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:13.150039 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:13.150065 (+ 26us) server_negotiation.cc:407] Connection header received
0927 04:02:13.150496 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:13.150502 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:13.150563 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:13.150746 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:13.151037 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:13.152118 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:13.153203 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:13.153624 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:13.153784 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:13.154596 (+ 812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:13.154600 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:13.154601 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:02:13.154854 (+ 253us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:13.154890 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:02:13.154901 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:13.154972 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:13.155073 (+ 101us) server_negotiation.cc:299] Negotiation successful
0927 04:02:13.155158 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":96,"threads_started":1}
I20260927 04:02:13.155354 1211 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90dcafd31f39443386594cc5480318ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90dcafd31f39443386594cc5480318ec" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 44297 } }
I20260927 04:02:13.155470 1211 leader_election.cc:302] T 00000000000000000000000000000000 P 90dcafd31f39443386594cc5480318ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90dcafd31f39443386594cc5480318ec; no voters:
I20260927 04:02:13.155669 1211 leader_election.cc:288] T 00000000000000000000000000000000 P 90dcafd31f39443386594cc5480318ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:02:13.155745 1216 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90dcafd31f39443386594cc5480318ec [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:02:13.155893 1216 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90dcafd31f39443386594cc5480318ec [term 1 LEADER]: Becoming Leader. State: Replica: 90dcafd31f39443386594cc5480318ec, State: Running, Role: LEADER
I20260927 04:02:13.155963 1211 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90dcafd31f39443386594cc5480318ec [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:02:13.156018 1216 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90dcafd31f39443386594cc5480318ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90dcafd31f39443386594cc5480318ec" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 44297 } }
I20260927 04:02:13.156407 1218 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90dcafd31f39443386594cc5480318ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90dcafd31f39443386594cc5480318ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90dcafd31f39443386594cc5480318ec" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 44297 } } }
I20260927 04:02:13.156569 1218 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90dcafd31f39443386594cc5480318ec [sys.catalog]: This master's current role is: LEADER
I20260927 04:02:13.156421 1217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90dcafd31f39443386594cc5480318ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90dcafd31f39443386594cc5480318ec. Latest consensus state: current_term: 1 leader_uuid: "90dcafd31f39443386594cc5480318ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90dcafd31f39443386594cc5480318ec" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 44297 } } }
I20260927 04:02:13.156842 1217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90dcafd31f39443386594cc5480318ec [sys.catalog]: This master's current role is: LEADER
I20260927 04:02:13.157024 1225 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:02:13.157543 1225 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:02:13.159127 1225 catalog_manager.cc:1347] Generated new cluster ID: f34f131cfc344cd4956080a6d941cfc4
I20260927 04:02:13.159184 1225 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:02:13.167171 1225 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:02:13.167889 1225 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:02:13.175557 1225 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 90dcafd31f39443386594cc5480318ec: Generated new TSK 0
I20260927 04:02:13.175796 1225 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 04:02:13.177448 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.65:0
--local_ip_for_outbound_sockets=127.15.185.65
--webserver_interface=127.15.185.65
--webserver_port=0
--tserver_master_addrs=127.15.185.126:44297
--builtin_ntp_servers=127.15.185.84:43239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 04:02:13.267870 1235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:02:13.268090 1235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:02:13.268113 1235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:02:13.269315 1235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:02:13.269358 1235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:02:13.269385 1235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.65
I20260927 04:02:13.270761 1235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:43239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.185.65
--webserver_port=0
--tserver_master_addrs=127.15.185.126:44297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.65
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:02:13.271003 1235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:02:13.271350 1235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:02:13.276876 1241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:13.276906 1244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:13.276909 1242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:13.277067 1235 server_base.cc:1034] running on GCE node
I20260927 04:02:13.277382 1235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:02:13.277673 1235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:02:13.278869 1235 hybrid_clock.cc:648] HybridClock initialized: now 1790481733278853 us; error 28 us; skew 500 ppm
I20260927 04:02:13.280465 1235 webserver.cc:460] Webserver started at http://127.15.185.65:38757/ using document root <none> and password file <none>
I20260927 04:02:13.280767 1235 fs_manager.cc:359] Metadata directory not provided
I20260927 04:02:13.280825 1235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:02:13.280966 1235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:02:13.282420 1235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/ts-0/data/instance:
uuid: "915adc013440481385d03cc396cefba3"
format_stamp: "Formatted at 2026-09-27 04:02:13 on dist-test-slave-glgz"
server_key: "d00a1c0db808e7089e45aa78bca146dd"
server_key_iv: "3426d8641fecca67636cbaeb736704b8"
server_key_version: "encryptionkey@0"
I20260927 04:02:13.282905 1235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/ts-0/wal/instance:
uuid: "915adc013440481385d03cc396cefba3"
format_stamp: "Formatted at 2026-09-27 04:02:13 on dist-test-slave-glgz"
server_key: "d00a1c0db808e7089e45aa78bca146dd"
server_key_iv: "3426d8641fecca67636cbaeb736704b8"
server_key_version: "encryptionkey@0"
I20260927 04:02:13.284665 1235 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260927 04:02:13.285567 1250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:13.285784 1235 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:02:13.285881 1235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/ts-0/wal
uuid: "915adc013440481385d03cc396cefba3"
format_stamp: "Formatted at 2026-09-27 04:02:13 on dist-test-slave-glgz"
server_key: "d00a1c0db808e7089e45aa78bca146dd"
server_key_iv: "3426d8641fecca67636cbaeb736704b8"
server_key_version: "encryptionkey@0"
I20260927 04:02:13.285953 1235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:02:13.323148 1235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:02:13.323503 1235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:02:13.323618 1235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:02:13.323838 1235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:02:13.324120 1235 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:02:13.324151 1235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:13.324175 1235 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:02:13.324187 1235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:13.330004 1235 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.65:36925
I20260927 04:02:13.330049 1364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.65:36925 every 8 connection(s)
I20260927 04:02:13.330399 1235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
I20260927 04:02:13.333755 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 1235
I20260927 04:02:13.333881 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/ts-0/wal/instance
I20260927 04:02:13.334079 16101 external_mini_cluster.cc:1428] Setting key fa2036279222cd22b46f8052968b6cf7
I20260927 04:02:13.334976 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.66:0
--local_ip_for_outbound_sockets=127.15.185.66
--webserver_interface=127.15.185.66
--webserver_port=0
--tserver_master_addrs=127.15.185.126:44297
--builtin_ntp_servers=127.15.185.84:43239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 04:02:13.336205 1214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:13.331218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:59493 (local address 127.15.185.126:44297)
0927 04:02:13.331283 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:13.331285 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:13.331557 (+ 272us) server_negotiation.cc:407] Connection header received
0927 04:02:13.332128 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:13.332130 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:13.332147 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:13.332246 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:13.332934 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:13.333450 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:13.334605 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:13.334807 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:13.334892 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:13.335907 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:13.335908 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:13.335909 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:02:13.335914 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:13.335928 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:02:13.335932 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:13.336002 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:13.336097 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 04:02:13.336123 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 04:02:13.336439 1367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:13.331234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:44297 (local address 127.15.185.65:59493)
0927 04:02:13.331429 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:13.331467 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:13.332018 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:13.332324 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:13.332328 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:13.332336 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:02:13.332837 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:13.332841 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:13.333598 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:13.333599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:13.334505 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:13.334508 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:13.334904 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:13.334905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:13.335035 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:13.335767 (+ 732us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 04:02:13.335768 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:02:13.335796 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 04:02:13.335818 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:02:13.336002 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:02:13.336009 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:02:13.336012 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 04:02:13.336100 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 04:02:13.336204 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":89,"threads_started":1}
I20260927 04:02:13.337128 1365 heartbeater.cc:347] Connected to a master server at 127.15.185.126:44297
I20260927 04:02:13.337265 1365 heartbeater.cc:464] Registering TS with master...
I20260927 04:02:13.337456 1365 heartbeater.cc:511] Master 127.15.185.126:44297 requested a full tablet report, sending...
I20260927 04:02:13.338024 1175 ts_manager.cc:194] Registered new tserver with Master: 915adc013440481385d03cc396cefba3 (127.15.185.65:36925)
I20260927 04:02:13.338745 1175 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.185.65:59493
W20260927 04:02:13.416615 1368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:02:13.416851 1368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:02:13.416874 1368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:02:13.418069 1368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:02:13.418114 1368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:02:13.418140 1368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.66
I20260927 04:02:13.419541 1368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:43239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.185.66
--webserver_port=0
--tserver_master_addrs=127.15.185.126:44297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.66
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:02:13.419808 1368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:02:13.420116 1368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:02:13.425366 1376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:13.425384 1373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:13.425375 1374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:13.425486 1368 server_base.cc:1034] running on GCE node
I20260927 04:02:13.425895 1368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:02:13.426116 1368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:02:13.427254 1368 hybrid_clock.cc:648] HybridClock initialized: now 1790481733427232 us; error 38 us; skew 500 ppm
I20260927 04:02:13.428630 1368 webserver.cc:460] Webserver started at http://127.15.185.66:40307/ using document root <none> and password file <none>
I20260927 04:02:13.428915 1368 fs_manager.cc:359] Metadata directory not provided
I20260927 04:02:13.428990 1368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:02:13.429141 1368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:02:13.430269 1368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/ts-1/data/instance:
uuid: "3271ae7011044bacac167c6db5537d63"
format_stamp: "Formatted at 2026-09-27 04:02:13 on dist-test-slave-glgz"
server_key: "def83da7756906ed004fd9b6c3165b88"
server_key_iv: "4f00e43e76348dd1e58fc086b7b62c6f"
server_key_version: "encryptionkey@0"
I20260927 04:02:13.430635 1368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/ts-1/wal/instance:
uuid: "3271ae7011044bacac167c6db5537d63"
format_stamp: "Formatted at 2026-09-27 04:02:13 on dist-test-slave-glgz"
server_key: "def83da7756906ed004fd9b6c3165b88"
server_key_iv: "4f00e43e76348dd1e58fc086b7b62c6f"
server_key_version: "encryptionkey@0"
I20260927 04:02:13.432049 1368 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 04:02:13.432842 1382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:13.433111 1368 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:02:13.433223 1368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/ts-1/wal
uuid: "3271ae7011044bacac167c6db5537d63"
format_stamp: "Formatted at 2026-09-27 04:02:13 on dist-test-slave-glgz"
server_key: "def83da7756906ed004fd9b6c3165b88"
server_key_iv: "4f00e43e76348dd1e58fc086b7b62c6f"
server_key_version: "encryptionkey@0"
I20260927 04:02:13.433296 1368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:02:13.462072 1368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:02:13.462455 1368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:02:13.462617 1368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:02:13.462867 1368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:02:13.463279 1368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:02:13.463336 1368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:13.463379 1368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:02:13.463407 1368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:13.469228 1368 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.66:33683
I20260927 04:02:13.469273 1496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.66:33683 every 8 connection(s)
I20260927 04:02:13.469597 1368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
I20260927 04:02:13.470608 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 1368
I20260927 04:02:13.470728 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/ts-1/wal/instance
I20260927 04:02:13.470932 16101 external_mini_cluster.cc:1428] Setting key f4d2178d5f432cc72a65f39ce93c71a2
I20260927 04:02:13.471900 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.67:0
--local_ip_for_outbound_sockets=127.15.185.67
--webserver_interface=127.15.185.67
--webserver_port=0
--tserver_master_addrs=127.15.185.126:44297
--builtin_ntp_servers=127.15.185.84:43239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 04:02:13.475255 1214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:13.470297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:41291 (local address 127.15.185.126:44297)
0927 04:02:13.470343 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:13.470346 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:13.470581 (+ 235us) server_negotiation.cc:407] Connection header received
0927 04:02:13.471310 (+ 729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:13.471312 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:13.471324 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:13.471369 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:13.472129 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:13.472630 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:13.473767 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:13.473967 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:13.474047 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:13.474930 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:13.474932 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:13.474932 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:02:13.474937 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:13.474957 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:02:13.474960 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:13.475019 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:13.475149 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 04:02:13.475175 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 04:02:13.475459 1499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:13.470267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:44297 (local address 127.15.185.66:41291)
0927 04:02:13.470458 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:13.470493 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:13.471202 (+ 709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:13.471434 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:13.471438 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:13.471447 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:02:13.471998 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:13.472006 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:13.472731 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:13.472733 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:13.473677 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:13.473679 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:13.474057 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:13.474058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:13.474196 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:13.474780 (+ 584us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 04:02:13.474781 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:02:13.474809 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 04:02:13.474833 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:02:13.475020 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:02:13.475026 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:02:13.475030 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 04:02:13.475147 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 04:02:13.475258 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":64,"threads_started":1}
I20260927 04:02:13.475908 1497 heartbeater.cc:347] Connected to a master server at 127.15.185.126:44297
I20260927 04:02:13.475997 1497 heartbeater.cc:464] Registering TS with master...
I20260927 04:02:13.476200 1497 heartbeater.cc:511] Master 127.15.185.126:44297 requested a full tablet report, sending...
I20260927 04:02:13.476632 1175 ts_manager.cc:194] Registered new tserver with Master: 3271ae7011044bacac167c6db5537d63 (127.15.185.66:33683)
I20260927 04:02:13.477193 1175 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.185.66:41291
W20260927 04:02:13.555606 1500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:02:13.555841 1500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:02:13.555864 1500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:02:13.557178 1500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:02:13.557224 1500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:02:13.557252 1500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.67
I20260927 04:02:13.558596 1500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:43239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.185.67
--webserver_port=0
--tserver_master_addrs=127.15.185.126:44297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.67
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:02:13.558794 1500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:02:13.559073 1500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:02:13.563894 1508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:13.563968 1506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:13.564018 1505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:13.564810 1500 server_base.cc:1034] running on GCE node
I20260927 04:02:13.565065 1500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:02:13.565330 1500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:02:13.566496 1500 hybrid_clock.cc:648] HybridClock initialized: now 1790481733566488 us; error 37 us; skew 500 ppm
I20260927 04:02:13.567790 1500 webserver.cc:460] Webserver started at http://127.15.185.67:40397/ using document root <none> and password file <none>
I20260927 04:02:13.568042 1500 fs_manager.cc:359] Metadata directory not provided
I20260927 04:02:13.568112 1500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:02:13.568262 1500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:02:13.569331 1500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/ts-2/data/instance:
uuid: "525519fdfe094c24a62815d9b3393026"
format_stamp: "Formatted at 2026-09-27 04:02:13 on dist-test-slave-glgz"
server_key: "c4ec95f6a65a24589018ad38bdb22110"
server_key_iv: "eae5fab4417e46b09fff00ce9175f996"
server_key_version: "encryptionkey@0"
I20260927 04:02:13.569691 1500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/ts-2/wal/instance:
uuid: "525519fdfe094c24a62815d9b3393026"
format_stamp: "Formatted at 2026-09-27 04:02:13 on dist-test-slave-glgz"
server_key: "c4ec95f6a65a24589018ad38bdb22110"
server_key_iv: "eae5fab4417e46b09fff00ce9175f996"
server_key_version: "encryptionkey@0"
I20260927 04:02:13.571025 1500 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 04:02:13.571800 1514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:13.571971 1500 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 04:02:13.572070 1500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/ts-2/wal
uuid: "525519fdfe094c24a62815d9b3393026"
format_stamp: "Formatted at 2026-09-27 04:02:13 on dist-test-slave-glgz"
server_key: "c4ec95f6a65a24589018ad38bdb22110"
server_key_iv: "eae5fab4417e46b09fff00ce9175f996"
server_key_version: "encryptionkey@0"
I20260927 04:02:13.572153 1500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:02:13.586467 1500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:02:13.586836 1500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:02:13.587007 1500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:02:13.587287 1500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:02:13.587662 1500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:02:13.587728 1500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:13.587766 1500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:02:13.587798 1500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:13.593904 1500 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.67:35165
I20260927 04:02:13.593950 1628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.67:35165 every 8 connection(s)
I20260927 04:02:13.594334 1500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
I20260927 04:02:13.596935 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 1500
I20260927 04:02:13.597065 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/ts-2/wal/instance
I20260927 04:02:13.597257 16101 external_mini_cluster.cc:1428] Setting key eec6bfdc8c700e72ba32871297980b3a
I20260927 04:02:13.599951 1214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:13.595114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:45625 (local address 127.15.185.126:44297)
0927 04:02:13.595187 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:13.595189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:13.595438 (+ 249us) server_negotiation.cc:407] Connection header received
0927 04:02:13.596021 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:13.596022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:13.596035 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:13.596086 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:13.596791 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:13.597424 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:13.598670 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:13.598859 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:13.598914 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:13.599735 (+ 821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:13.599737 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:13.599737 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:02:13.599741 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:13.599754 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:02:13.599757 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:13.599805 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:13.599871 (+ 66us) server_negotiation.cc:299] Negotiation successful
0927 04:02:13.599892 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 04:02:13.600157 1631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:13.595122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:44297 (local address 127.15.185.67:45625)
0927 04:02:13.595319 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:13.595353 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:13.595901 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:13.596161 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:13.596164 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:13.596174 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:02:13.596678 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:13.596683 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:13.597540 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:13.597541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:13.598592 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:13.598595 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:13.598935 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:13.598936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:13.599136 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:13.599608 (+ 472us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 04:02:13.599609 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:02:13.599637 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 04:02:13.599660 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:02:13.599808 (+ 148us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:02:13.599811 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:02:13.599813 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 04:02:13.599861 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 04:02:13.599937 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":84,"threads_started":1}
I20260927 04:02:13.600594 1629 heartbeater.cc:347] Connected to a master server at 127.15.185.126:44297
I20260927 04:02:13.600708 1629 heartbeater.cc:464] Registering TS with master...
I20260927 04:02:13.600888 1629 heartbeater.cc:511] Master 127.15.185.126:44297 requested a full tablet report, sending...
I20260927 04:02:13.601248 1175 ts_manager.cc:194] Registered new tserver with Master: 525519fdfe094c24a62815d9b3393026 (127.15.185.67:35165)
I20260927 04:02:13.601617 1175 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.185.67:45625
I20260927 04:02:13.608888 16101 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 04:02:13.613302 1214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:13.610296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58700 (local address 127.15.185.126:44297)
0927 04:02:13.610369 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:13.610372 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:13.610453 (+ 81us) server_negotiation.cc:407] Connection header received
0927 04:02:13.610544 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:13.610545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:13.610556 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:13.610614 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:13.610783 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:13.611340 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:13.612103 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:13.612297 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:13.612386 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:13.613034 (+ 648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:13.613036 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:13.613036 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:02:13.613041 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:13.613055 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:02:13.613059 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:13.613111 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:13.613192 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 04:02:13.613214 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 04:02:13.614729 1175 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58700:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 04:02:13.616387 1175 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 04:02:13.622944 1642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:13.619065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36968 (local address 127.15.185.67:35165)
0927 04:02:13.619209 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:13.619214 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:13.619225 (+ 11us) server_negotiation.cc:407] Connection header received
0927 04:02:13.619266 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:13.619269 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:13.619285 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:13.619543 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:02:13.619605 (+ 62us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:13.620403 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:13.621849 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:13.622529 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:13.622621 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:13.622722 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:13.622813 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 04:02:13.622851 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":82,"threads_started":1}
I20260927 04:02:13.622944 1640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:13.618749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.67:35165 (local address 127.0.0.1:36968)
0927 04:02:13.619012 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:13.619023 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:13.619088 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:13.619384 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:13.619385 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:13.619386 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:02:13.619458 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:13.619460 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:13.620488 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:13.620489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:13.621647 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:13.621649 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:13.622620 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:13.622621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:13.622698 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:13.622720 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 04:02:13.622797 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 04:02:13.622853 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":2,"thread_start_us":38,"threads_started":1}
I20260927 04:02:13.623301 1637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:13.618538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.65:36925 (local address 127.0.0.1:38408)
0927 04:02:13.618654 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:13.618701 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:13.618763 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:13.619360 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:13.619363 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:13.619366 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:02:13.619526 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:13.619528 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:13.620823 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:13.620824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:13.622144 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:13.622146 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:13.623034 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:13.623035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:13.623126 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:13.623131 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 04:02:13.623163 (+ 32us) client_negotiation.cc:240] Negotiation successful
0927 04:02:13.623220 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"mutex_wait_us":144,"thread_start_us":70,"threads_started":1}
I20260927 04:02:13.623625 1638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:13.618893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.66:33683 (local address 127.0.0.1:52228)
0927 04:02:13.619452 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:13.619462 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:13.619510 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:13.619716 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:13.619717 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:13.619719 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:02:13.619760 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:13.619761 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:13.620939 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:13.620941 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:13.621992 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:13.621994 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:13.623412 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:13.623413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:13.623471 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:13.623474 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 04:02:13.623530 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 04:02:13.623567 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":79,"threads_started":1}
I20260927 04:02:13.623801 1641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:13.619002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52228 (local address 127.15.185.66:33683)
0927 04:02:13.619281 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:13.619285 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:13.619566 (+ 281us) server_negotiation.cc:407] Connection header received
0927 04:02:13.619609 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:13.619611 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:13.619628 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:13.619701 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:02:13.619820 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:13.620859 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:13.622075 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:13.622832 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:13.622904 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:13.622998 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:13.623647 (+ 649us) server_negotiation.cc:299] Negotiation successful
0927 04:02:13.623699 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":116,"threads_started":1}
I20260927 04:02:13.623880 1562 tablet_service.cc:1467] Processing CreateTablet for tablet 156d124ce18b402395084559cdcccce7 (DEFAULT_TABLE table=test-table [id=c9e0ee56b0974623a518298c2674a299]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:02:13.624269 1562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 156d124ce18b402395084559cdcccce7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:02:13.624372 1430 tablet_service.cc:1467] Processing CreateTablet for tablet 156d124ce18b402395084559cdcccce7 (DEFAULT_TABLE table=test-table [id=c9e0ee56b0974623a518298c2674a299]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:02:13.624629 1430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 156d124ce18b402395084559cdcccce7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:02:13.627067 1643 tablet_bootstrap.cc:492] T 156d124ce18b402395084559cdcccce7 P 525519fdfe094c24a62815d9b3393026: Bootstrap starting.
I20260927 04:02:13.627348 1639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:13.618655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38408 (local address 127.15.185.65:36925)
0927 04:02:13.619175 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:13.619180 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:13.619192 (+ 12us) server_negotiation.cc:407] Connection header received
0927 04:02:13.619236 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:13.619238 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:13.619255 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:13.619352 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:02:13.619589 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:13.620723 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:13.622232 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:13.622941 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:13.627071 (+ 4130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:13.627181 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:13.627208 (+ 27us) server_negotiation.cc:299] Negotiation successful
0927 04:02:13.627260 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":158,"threads_started":1}
I20260927 04:02:13.627938 1643 tablet_bootstrap.cc:654] T 156d124ce18b402395084559cdcccce7 P 525519fdfe094c24a62815d9b3393026: Neither blocks nor log segments found. Creating new log.
I20260927 04:02:13.627980 1298 tablet_service.cc:1467] Processing CreateTablet for tablet 156d124ce18b402395084559cdcccce7 (DEFAULT_TABLE table=test-table [id=c9e0ee56b0974623a518298c2674a299]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:02:13.628283 1643 log.cc:824] T 156d124ce18b402395084559cdcccce7 P 525519fdfe094c24a62815d9b3393026: Log is configured to *not* fsync() on all Append() calls
I20260927 04:02:13.628325 1298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 156d124ce18b402395084559cdcccce7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:02:13.628594 1644 tablet_bootstrap.cc:492] T 156d124ce18b402395084559cdcccce7 P 3271ae7011044bacac167c6db5537d63: Bootstrap starting.
I20260927 04:02:13.629113 1643 tablet_bootstrap.cc:492] T 156d124ce18b402395084559cdcccce7 P 525519fdfe094c24a62815d9b3393026: No bootstrap required, opened a new log
I20260927 04:02:13.629190 1643 ts_tablet_manager.cc:1397] T 156d124ce18b402395084559cdcccce7 P 525519fdfe094c24a62815d9b3393026: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 04:02:13.629264 1644 tablet_bootstrap.cc:654] T 156d124ce18b402395084559cdcccce7 P 3271ae7011044bacac167c6db5537d63: Neither blocks nor log segments found. Creating new log.
I20260927 04:02:13.629566 1644 log.cc:824] T 156d124ce18b402395084559cdcccce7 P 3271ae7011044bacac167c6db5537d63: Log is configured to *not* fsync() on all Append() calls
I20260927 04:02:13.630298 1644 tablet_bootstrap.cc:492] T 156d124ce18b402395084559cdcccce7 P 3271ae7011044bacac167c6db5537d63: No bootstrap required, opened a new log
I20260927 04:02:13.630370 1644 ts_tablet_manager.cc:1397] T 156d124ce18b402395084559cdcccce7 P 3271ae7011044bacac167c6db5537d63: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 04:02:13.631585 1643 raft_consensus.cc:348] T 156d124ce18b402395084559cdcccce7 P 525519fdfe094c24a62815d9b3393026 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "915adc013440481385d03cc396cefba3" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 36925 } } peers { permanent_uuid: "525519fdfe094c24a62815d9b3393026" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 35165 } } peers { permanent_uuid: "3271ae7011044bacac167c6db5537d63" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 33683 } }
I20260927 04:02:13.631740 1643 raft_consensus.cc:374] T 156d124ce18b402395084559cdcccce7 P 525519fdfe094c24a62815d9b3393026 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:02:13.631784 1643 raft_consensus.cc:729] T 156d124ce18b402395084559cdcccce7 P 525519fdfe094c24a62815d9b3393026 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525519fdfe094c24a62815d9b3393026, State: Initialized, Role: FOLLOWER
I20260927 04:02:13.631898 1643 consensus_queue.cc:260] T 156d124ce18b402395084559cdcccce7 P 525519fdfe094c24a62815d9b3393026 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "915adc013440481385d03cc396cefba3" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 36925 } } peers { permanent_uuid: "525519fdfe094c24a62815d9b3393026" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 35165 } } peers { permanent_uuid: "3271ae7011044bacac167c6db5537d63" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 33683 } }
I20260927 04:02:13.632175 1643 ts_tablet_manager.cc:1428] T 156d124ce18b402395084559cdcccce7 P 525519fdfe094c24a62815d9b3393026: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 04:02:13.632201 1629 heartbeater.cc:503] Master 127.15.185.126:44297 was elected leader, sending a full tablet report...
I20260927 04:02:13.632225 1647 tablet_bootstrap.cc:492] T 156d124ce18b402395084559cdcccce7 P 915adc013440481385d03cc396cefba3: Bootstrap starting.
I20260927 04:02:13.633034 1644 raft_consensus.cc:348] T 156d124ce18b402395084559cdcccce7 P 3271ae7011044bacac167c6db5537d63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "915adc013440481385d03cc396cefba3" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 36925 } } peers { permanent_uuid: "525519fdfe094c24a62815d9b3393026" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 35165 } } peers { permanent_uuid: "3271ae7011044bacac167c6db5537d63" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 33683 } }
I20260927 04:02:13.633203 1644 raft_consensus.cc:374] T 156d124ce18b402395084559cdcccce7 P 3271ae7011044bacac167c6db5537d63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:02:13.633247 1644 raft_consensus.cc:729] T 156d124ce18b402395084559cdcccce7 P 3271ae7011044bacac167c6db5537d63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3271ae7011044bacac167c6db5537d63, State: Initialized, Role: FOLLOWER
I20260927 04:02:13.633425 1647 tablet_bootstrap.cc:654] T 156d124ce18b402395084559cdcccce7 P 915adc013440481385d03cc396cefba3: Neither blocks nor log segments found. Creating new log.
I20260927 04:02:13.633374 1644 consensus_queue.cc:260] T 156d124ce18b402395084559cdcccce7 P 3271ae7011044bacac167c6db5537d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "915adc013440481385d03cc396cefba3" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 36925 } } peers { permanent_uuid: "525519fdfe094c24a62815d9b3393026" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 35165 } } peers { permanent_uuid: "3271ae7011044bacac167c6db5537d63" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 33683 } }
I20260927 04:02:13.633661 1644 ts_tablet_manager.cc:1428] T 156d124ce18b402395084559cdcccce7 P 3271ae7011044bacac167c6db5537d63: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260927 04:02:13.633699 1497 heartbeater.cc:503] Master 127.15.185.126:44297 was elected leader, sending a full tablet report...
I20260927 04:02:13.633769 1647 log.cc:824] T 156d124ce18b402395084559cdcccce7 P 915adc013440481385d03cc396cefba3: Log is configured to *not* fsync() on all Append() calls
I20260927 04:02:13.634460 1647 tablet_bootstrap.cc:492] T 156d124ce18b402395084559cdcccce7 P 915adc013440481385d03cc396cefba3: No bootstrap required, opened a new log
I20260927 04:02:13.634531 1647 ts_tablet_manager.cc:1397] T 156d124ce18b402395084559cdcccce7 P 915adc013440481385d03cc396cefba3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 04:02:13.636565 1647 raft_consensus.cc:348] T 156d124ce18b402395084559cdcccce7 P 915adc013440481385d03cc396cefba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "915adc013440481385d03cc396cefba3" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 36925 } } peers { permanent_uuid: "525519fdfe094c24a62815d9b3393026" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 35165 } } peers { permanent_uuid: "3271ae7011044bacac167c6db5537d63" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 33683 } }
I20260927 04:02:13.636685 1647 raft_consensus.cc:374] T 156d124ce18b402395084559cdcccce7 P 915adc013440481385d03cc396cefba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:02:13.636708 1647 raft_consensus.cc:729] T 156d124ce18b402395084559cdcccce7 P 915adc013440481385d03cc396cefba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 915adc013440481385d03cc396cefba3, State: Initialized, Role: FOLLOWER
I20260927 04:02:13.636822 1647 consensus_queue.cc:260] T 156d124ce18b402395084559cdcccce7 P 915adc013440481385d03cc396cefba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "915adc013440481385d03cc396cefba3" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 36925 } } peers { permanent_uuid: "525519fdfe094c24a62815d9b3393026" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 35165 } } peers { permanent_uuid: "3271ae7011044bacac167c6db5537d63" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 33683 } }
I20260927 04:02:13.637007 1647 ts_tablet_manager.cc:1428] T 156d124ce18b402395084559cdcccce7 P 915adc013440481385d03cc396cefba3: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 04:02:13.637115 1365 heartbeater.cc:503] Master 127.15.185.126:44297 was elected leader, sending a full tablet report...
W20260927 04:02:13.720132 1498 tablet.cc:2369] T 156d124ce18b402395084559cdcccce7 P 3271ae7011044bacac167c6db5537d63: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 04:02:13.831146 1366 tablet.cc:2369] T 156d124ce18b402395084559cdcccce7 P 915adc013440481385d03cc396cefba3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 04:02:13.844923 1630 tablet.cc:2369] T 156d124ce18b402395084559cdcccce7 P 525519fdfe094c24a62815d9b3393026: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 04:02:13.886902 1651 raft_consensus.cc:482] T 156d124ce18b402395084559cdcccce7 P 915adc013440481385d03cc396cefba3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:02:13.887097 1651 raft_consensus.cc:504] T 156d124ce18b402395084559cdcccce7 P 915adc013440481385d03cc396cefba3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "915adc013440481385d03cc396cefba3" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 36925 } } peers { permanent_uuid: "525519fdfe094c24a62815d9b3393026" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 35165 } } peers { permanent_uuid: "3271ae7011044bacac167c6db5537d63" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 33683 } }
I20260927 04:02:13.887491 1651 leader_election.cc:288] T 156d124ce18b402395084559cdcccce7 P 915adc013440481385d03cc396cefba3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 525519fdfe094c24a62815d9b3393026 (127.15.185.67:35165), 3271ae7011044bacac167c6db5537d63 (127.15.185.66:33683)
I20260927 04:02:13.890511 1642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:13.887755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:55649 (local address 127.15.185.67:35165)
0927 04:02:13.887820 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:13.887822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:13.887896 (+ 74us) server_negotiation.cc:407] Connection header received
0927 04:02:13.887990 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:13.887991 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:13.888002 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:13.888052 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:02:13.888239 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:13.888715 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:13.889716 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:13.890156 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:13.890237 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:13.890246 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:13.890387 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 04:02:13.890436 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 04:02:13.890522 1652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:13.887685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.67:35165 (local address 127.15.185.65:55649)
0927 04:02:13.887819 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:13.887832 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:13.887897 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:13.888080 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:13.888082 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:13.888084 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:02:13.888157 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:13.888159 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:13.888801 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:13.888802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:13.889598 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:13.889600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:13.890232 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:13.890233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:13.890307 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:13.890310 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 04:02:13.890361 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 04:02:13.890390 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":2,"thread_start_us":90,"threads_started":1}
I20260927 04:02:13.890560 1641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:13.887981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:49041 (local address 127.15.185.66:33683)
0927 04:02:13.888057 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:13.888059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:13.888069 (+ 10us) server_negotiation.cc:407] Connection header received
0927 04:02:13.888117 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:13.888118 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:13.888133 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:13.888196 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:02:13.888354 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:13.888803 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:13.889731 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:13.890161 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:13.890256 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:13.890264 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:13.890442 (+ 178us) server_negotiation.cc:299] Negotiation successful
0927 04:02:13.890486 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 04:02:13.890776 1653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:13.887896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.66:33683 (local address 127.15.185.65:49041)
0927 04:02:13.887985 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:13.888019 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:13.888062 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:13.888198 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:13.888199 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:13.888201 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:02:13.888283 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:13.888285 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:13.888885 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:13.888886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:13.889634 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:13.889635 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:13.890278 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:13.890279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:13.890363 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:13.890367 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 04:02:13.890425 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 04:02:13.890460 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"mutex_wait_us":1,"thread_start_us":40,"threads_started":1}
I20260927 04:02:13.891230 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156d124ce18b402395084559cdcccce7" candidate_uuid: "915adc013440481385d03cc396cefba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525519fdfe094c24a62815d9b3393026" is_pre_election: true
I20260927 04:02:13.891228 1451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156d124ce18b402395084559cdcccce7" candidate_uuid: "915adc013440481385d03cc396cefba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3271ae7011044bacac167c6db5537d63" is_pre_election: true
I20260927 04:02:13.891390 1583 raft_consensus.cc:2454] T 156d124ce18b402395084559cdcccce7 P 525519fdfe094c24a62815d9b3393026 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 915adc013440481385d03cc396cefba3 in term 0.
I20260927 04:02:13.891390 1451 raft_consensus.cc:2454] T 156d124ce18b402395084559cdcccce7 P 3271ae7011044bacac167c6db5537d63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 915adc013440481385d03cc396cefba3 in term 0.
I20260927 04:02:13.891615 1253 leader_election.cc:302] T 156d124ce18b402395084559cdcccce7 P 915adc013440481385d03cc396cefba3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3271ae7011044bacac167c6db5537d63, 915adc013440481385d03cc396cefba3; no voters:
I20260927 04:02:13.891793 1651 raft_consensus.cc:2781] T 156d124ce18b402395084559cdcccce7 P 915adc013440481385d03cc396cefba3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 04:02:13.891865 1651 raft_consensus.cc:482] T 156d124ce18b402395084559cdcccce7 P 915adc013440481385d03cc396cefba3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 04:02:13.891882 1651 raft_consensus.cc:3037] T 156d124ce18b402395084559cdcccce7 P 915adc013440481385d03cc396cefba3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:02:13.892617 1651 raft_consensus.cc:504] T 156d124ce18b402395084559cdcccce7 P 915adc013440481385d03cc396cefba3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "915adc013440481385d03cc396cefba3" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 36925 } } peers { permanent_uuid: "525519fdfe094c24a62815d9b3393026" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 35165 } } peers { permanent_uuid: "3271ae7011044bacac167c6db5537d63" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 33683 } }
I20260927 04:02:13.892755 1651 leader_election.cc:288] T 156d124ce18b402395084559cdcccce7 P 915adc013440481385d03cc396cefba3 [CANDIDATE]: Term 1 election: Requested vote from peers 525519fdfe094c24a62815d9b3393026 (127.15.185.67:35165), 3271ae7011044bacac167c6db5537d63 (127.15.185.66:33683)
I20260927 04:02:13.892941 1583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156d124ce18b402395084559cdcccce7" candidate_uuid: "915adc013440481385d03cc396cefba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "525519fdfe094c24a62815d9b3393026"
I20260927 04:02:13.892956 1451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156d124ce18b402395084559cdcccce7" candidate_uuid: "915adc013440481385d03cc396cefba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3271ae7011044bacac167c6db5537d63"
I20260927 04:02:13.893028 1451 raft_consensus.cc:3037] T 156d124ce18b402395084559cdcccce7 P 3271ae7011044bacac167c6db5537d63 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:02:13.893028 1583 raft_consensus.cc:3037] T 156d124ce18b402395084559cdcccce7 P 525519fdfe094c24a62815d9b3393026 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:02:13.893944 1451 raft_consensus.cc:2454] T 156d124ce18b402395084559cdcccce7 P 3271ae7011044bacac167c6db5537d63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 915adc013440481385d03cc396cefba3 in term 1.
I20260927 04:02:13.893971 1583 raft_consensus.cc:2454] T 156d124ce18b402395084559cdcccce7 P 525519fdfe094c24a62815d9b3393026 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 915adc013440481385d03cc396cefba3 in term 1.
I20260927 04:02:13.894097 1253 leader_election.cc:302] T 156d124ce18b402395084559cdcccce7 P 915adc013440481385d03cc396cefba3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3271ae7011044bacac167c6db5537d63, 915adc013440481385d03cc396cefba3; no voters:
I20260927 04:02:13.894199 1651 raft_consensus.cc:2781] T 156d124ce18b402395084559cdcccce7 P 915adc013440481385d03cc396cefba3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:02:13.894338 1651 raft_consensus.cc:686] T 156d124ce18b402395084559cdcccce7 P 915adc013440481385d03cc396cefba3 [term 1 LEADER]: Becoming Leader. State: Replica: 915adc013440481385d03cc396cefba3, State: Running, Role: LEADER
I20260927 04:02:13.894436 1651 consensus_queue.cc:237] T 156d124ce18b402395084559cdcccce7 P 915adc013440481385d03cc396cefba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "915adc013440481385d03cc396cefba3" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 36925 } } peers { permanent_uuid: "525519fdfe094c24a62815d9b3393026" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 35165 } } peers { permanent_uuid: "3271ae7011044bacac167c6db5537d63" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 33683 } }
I20260927 04:02:13.895269 1175 catalog_manager.cc:5495] T 156d124ce18b402395084559cdcccce7 P 915adc013440481385d03cc396cefba3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 915adc013440481385d03cc396cefba3 (127.15.185.65). New cstate: current_term: 1 leader_uuid: "915adc013440481385d03cc396cefba3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "915adc013440481385d03cc396cefba3" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 36925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "525519fdfe094c24a62815d9b3393026" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 35165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3271ae7011044bacac167c6db5537d63" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 33683 } health_report { overall_health: UNKNOWN } } }
I20260927 04:02:13.906122 1639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:13.903726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38416 (local address 127.15.185.65:36925)
0927 04:02:13.903795 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:13.903797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:13.903806 (+ 9us) server_negotiation.cc:407] Connection header received
0927 04:02:13.903848 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:13.903849 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:13.903871 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:13.903960 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:02:13.904112 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:13.904587 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:13.905343 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:13.905545 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:13.905622 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:13.905792 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:02:13.905903 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:02:13.905998 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:13.906017 (+ 19us) server_negotiation.cc:299] Negotiation successful
0927 04:02:13.906039 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 04:02:13.907979 1451 raft_consensus.cc:1261] T 156d124ce18b402395084559cdcccce7 P 3271ae7011044bacac167c6db5537d63 [term 1 FOLLOWER]: Refusing update from remote peer 915adc013440481385d03cc396cefba3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:02:13.907992 1583 raft_consensus.cc:1261] T 156d124ce18b402395084559cdcccce7 P 525519fdfe094c24a62815d9b3393026 [term 1 FOLLOWER]: Refusing update from remote peer 915adc013440481385d03cc396cefba3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:02:13.908387 1654 consensus_queue.cc:1035] T 156d124ce18b402395084559cdcccce7 P 915adc013440481385d03cc396cefba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "525519fdfe094c24a62815d9b3393026" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 35165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:02:13.908520 1651 consensus_queue.cc:1035] T 156d124ce18b402395084559cdcccce7 P 915adc013440481385d03cc396cefba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3271ae7011044bacac167c6db5537d63" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 33683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:02:13.910969 1657 mvcc.cc:205] Tried to move back new op lower bound from 7333813182084321280 to 7333813182032138240. Current Snapshot: MvccSnapshot[applied={T|T < 7333813182084321280}]
I20260927 04:02:13.911348 1658 mvcc.cc:205] Tried to move back new op lower bound from 7333813182084321280 to 7333813182032138240. Current Snapshot: MvccSnapshot[applied={T|T < 7333813182084321280}]
I20260927 04:02:13.913213 1174 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:02:13.913304 1174 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:02:13.914131 1174 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 90dcafd31f39443386594cc5480318ec: Sending DeleteTablet for 3 replicas of tablet 156d124ce18b402395084559cdcccce7
I20260927 04:02:13.914489 1298 tablet_service.cc:1514] Processing DeleteTablet for tablet 156d124ce18b402395084559cdcccce7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:02:13 UTC) from {username='slave'} at 127.0.0.1:38408
I20260927 04:02:13.914475 1562 tablet_service.cc:1514] Processing DeleteTablet for tablet 156d124ce18b402395084559cdcccce7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:02:13 UTC) from {username='slave'} at 127.0.0.1:36968
I20260927 04:02:13.914727 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 1235
I20260927 04:02:13.914737 1430 tablet_service.cc:1514] Processing DeleteTablet for tablet 156d124ce18b402395084559cdcccce7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:02:13 UTC) from {username='slave'} at 127.0.0.1:52228
I20260927 04:02:13.915520 1669 tablet_replica.cc:330] T 156d124ce18b402395084559cdcccce7 P 3271ae7011044bacac167c6db5537d63: stopping tablet replica
I20260927 04:02:13.915715 1669 raft_consensus.cc:2229] T 156d124ce18b402395084559cdcccce7 P 3271ae7011044bacac167c6db5537d63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:02:13.915827 1669 raft_consensus.cc:2258] T 156d124ce18b402395084559cdcccce7 P 3271ae7011044bacac167c6db5537d63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:02:13.916291 1669 ts_tablet_manager.cc:1905] T 156d124ce18b402395084559cdcccce7 P 3271ae7011044bacac167c6db5537d63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:02:13.917634 1668 tablet_replica.cc:330] T 156d124ce18b402395084559cdcccce7 P 525519fdfe094c24a62815d9b3393026: stopping tablet replica
I20260927 04:02:13.917747 1668 raft_consensus.cc:2229] T 156d124ce18b402395084559cdcccce7 P 525519fdfe094c24a62815d9b3393026 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:02:13.917855 1668 raft_consensus.cc:2258] T 156d124ce18b402395084559cdcccce7 P 525519fdfe094c24a62815d9b3393026 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:02:13.918256 1668 ts_tablet_manager.cc:1905] T 156d124ce18b402395084559cdcccce7 P 525519fdfe094c24a62815d9b3393026: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:02:13.918496 1669 ts_tablet_manager.cc:1918] T 156d124ce18b402395084559cdcccce7 P 3271ae7011044bacac167c6db5537d63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:02:13.918582 1669 log.cc:1196] T 156d124ce18b402395084559cdcccce7 P 3271ae7011044bacac167c6db5537d63: Deleting WAL directory at /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/ts-1/wal/wals/156d124ce18b402395084559cdcccce7
I20260927 04:02:13.918839 1669 ts_tablet_manager.cc:1939] T 156d124ce18b402395084559cdcccce7 P 3271ae7011044bacac167c6db5537d63: Deleting consensus metadata
I20260927 04:02:13.919667 1161 catalog_manager.cc:4841] TS 3271ae7011044bacac167c6db5537d63 (127.15.185.66:33683): tablet 156d124ce18b402395084559cdcccce7 (table test-table [id=c9e0ee56b0974623a518298c2674a299]) successfully deleted
I20260927 04:02:13.920025 1668 ts_tablet_manager.cc:1918] T 156d124ce18b402395084559cdcccce7 P 525519fdfe094c24a62815d9b3393026: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:02:13.920110 1668 log.cc:1196] T 156d124ce18b402395084559cdcccce7 P 525519fdfe094c24a62815d9b3393026: Deleting WAL directory at /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481498018407-16101-0/minicluster-data/ts-2/wal/wals/156d124ce18b402395084559cdcccce7
I20260927 04:02:13.920329 1668 ts_tablet_manager.cc:1939] T 156d124ce18b402395084559cdcccce7 P 525519fdfe094c24a62815d9b3393026: Deleting consensus metadata
I20260927 04:02:13.920733 1161 catalog_manager.cc:4841] TS 525519fdfe094c24a62815d9b3393026 (127.15.185.67:35165): tablet 156d124ce18b402395084559cdcccce7 (table test-table [id=c9e0ee56b0974623a518298c2674a299]) successfully deleted
W20260927 04:02:13.921955 1161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.185.65:36925 (error 108)
I20260927 04:02:13.922155 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 1368
I20260927 04:02:13.922997 1638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:13.922864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.65:36925 (local address 127.0.0.1:38418)
0927 04:02:13.922914 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:13.922955 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.65:36925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260927 04:02:13.923147 1161 catalog_manager.cc:4584] TS 915adc013440481385d03cc396cefba3 (127.15.185.65:36925): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 156d124ce18b402395084559cdcccce7: Network error: Client connection negotiation failed: client connection to 127.15.185.65:36925: connect: Connection refused (error 111)
I20260927 04:02:13.926949 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 1500
I20260927 04:02:13.931270 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 1144
2026-09-27T04:02:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (944 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:02:13 dist-test-slave-glgz krb5kdc[1673](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:02:13 dist-test-slave-glgz krb5kdc[1673](info): set up 2 sockets
Sep 27 04:02:13 dist-test-slave-glgz krb5kdc[1673](info): commencing operation
krb5kdc: starting...
W20260927 04:02:15.015998 16101 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Sep 27 04:02:15 dist-test-slave-glgz krb5kdc[1673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481735, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:02:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:02:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.185.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab.
Entry for principal kudu/127.15.185.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab.
I20260927 04:02:15.142402 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:33325
--webserver_interface=127.15.185.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:33103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.185.126:33325
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:02:15.221990 1687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:02:15.222244 1687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:02:15.222281 1687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:02:15.223440 1687 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:02:15.223505 1687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:02:15.223528 1687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:02:15.223549 1687 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:02:15.223577 1687 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:02:15.225050 1687 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:33103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.185.126:33325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:33325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:02:15.225324 1687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:02:15.225545 1687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:02:15.230578 1695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:15.230626 1693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:15.230693 1692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:15.230834 1687 server_base.cc:1034] running on GCE node
I20260927 04:02:15.231153 1687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:02:15.231457 1687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:02:15.232687 1687 hybrid_clock.cc:648] HybridClock initialized: now 1790481735232711 us; error 68 us; skew 500 ppm
Sep 27 04:02:15 dist-test-slave-glgz krb5kdc[1673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481735, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:02:15.245992 1687 init.cc:377] Logged in from keytab as kudu/127.15.185.126@KRBTEST.COM (short username kudu)
I20260927 04:02:15.246528 1687 webserver.cc:460] Webserver started at http://127.15.185.126:35661/ using document root <none> and password file <none>
I20260927 04:02:15.246780 1687 fs_manager.cc:359] Metadata directory not provided
I20260927 04:02:15.246850 1687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:02:15.246997 1687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:02:15.248157 1687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/minicluster-data/master-0/data/instance:
uuid: "31ff61a46df749a7bd401859fcc13938"
format_stamp: "Formatted at 2026-09-27 04:02:15 on dist-test-slave-glgz"
server_key: "ed00873d9c7b104d3161389b756697bc"
server_key_iv: "b63133b32f1bc8fb279112c911f28481"
server_key_version: "encryptionkey@0"
I20260927 04:02:15.248557 1687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/minicluster-data/master-0/wal/instance:
uuid: "31ff61a46df749a7bd401859fcc13938"
format_stamp: "Formatted at 2026-09-27 04:02:15 on dist-test-slave-glgz"
server_key: "ed00873d9c7b104d3161389b756697bc"
server_key_iv: "b63133b32f1bc8fb279112c911f28481"
server_key_version: "encryptionkey@0"
I20260927 04:02:15.250182 1687 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 04:02:15.251243 1702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:15.251451 1687 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:02:15.251555 1687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/minicluster-data/master-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/minicluster-data/master-0/wal
uuid: "31ff61a46df749a7bd401859fcc13938"
format_stamp: "Formatted at 2026-09-27 04:02:15 on dist-test-slave-glgz"
server_key: "ed00873d9c7b104d3161389b756697bc"
server_key_iv: "b63133b32f1bc8fb279112c911f28481"
server_key_version: "encryptionkey@0"
I20260927 04:02:15.251632 1687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:02:15.274946 1687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:02:15.276921 1687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:02:15.277082 1687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:02:15.281282 1687 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.126:33325
I20260927 04:02:15.281335 1754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.126:33325 every 8 connection(s)
I20260927 04:02:15.281702 1687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
I20260927 04:02:15.282531 1755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:02:15.284847 1755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31ff61a46df749a7bd401859fcc13938: Bootstrap starting.
I20260927 04:02:15.285441 1755 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31ff61a46df749a7bd401859fcc13938: Neither blocks nor log segments found. Creating new log.
I20260927 04:02:15.285735 1755 log.cc:824] T 00000000000000000000000000000000 P 31ff61a46df749a7bd401859fcc13938: Log is configured to *not* fsync() on all Append() calls
I20260927 04:02:15.286408 1755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31ff61a46df749a7bd401859fcc13938: No bootstrap required, opened a new log
I20260927 04:02:15.287375 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 1687
I20260927 04:02:15.287503 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/minicluster-data/master-0/wal/instance
I20260927 04:02:15.287735 16101 external_mini_cluster.cc:1428] Setting key c72aad17b6513a671b4b12b15f4cbd96
I20260927 04:02:15.288710 1755 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31ff61a46df749a7bd401859fcc13938 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31ff61a46df749a7bd401859fcc13938" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 33325 } }
I20260927 04:02:15.288887 1755 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31ff61a46df749a7bd401859fcc13938 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:02:15.288945 1755 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31ff61a46df749a7bd401859fcc13938 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31ff61a46df749a7bd401859fcc13938, State: Initialized, Role: FOLLOWER
I20260927 04:02:15.289083 1755 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31ff61a46df749a7bd401859fcc13938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31ff61a46df749a7bd401859fcc13938" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 33325 } }
I20260927 04:02:15.289170 1755 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31ff61a46df749a7bd401859fcc13938 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:02:15.289214 1755 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31ff61a46df749a7bd401859fcc13938 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:02:15.289269 1755 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31ff61a46df749a7bd401859fcc13938 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:02:15.290221 1755 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31ff61a46df749a7bd401859fcc13938 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31ff61a46df749a7bd401859fcc13938" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 33325 } }
I20260927 04:02:15.290347 1755 leader_election.cc:302] T 00000000000000000000000000000000 P 31ff61a46df749a7bd401859fcc13938 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31ff61a46df749a7bd401859fcc13938; no voters:
I20260927 04:02:15.290511 1755 leader_election.cc:288] T 00000000000000000000000000000000 P 31ff61a46df749a7bd401859fcc13938 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:02:15.290545 1760 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31ff61a46df749a7bd401859fcc13938 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:02:15.290735 1760 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31ff61a46df749a7bd401859fcc13938 [term 1 LEADER]: Becoming Leader. State: Replica: 31ff61a46df749a7bd401859fcc13938, State: Running, Role: LEADER
I20260927 04:02:15.290776 1755 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31ff61a46df749a7bd401859fcc13938 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:02:15.290838 1760 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31ff61a46df749a7bd401859fcc13938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31ff61a46df749a7bd401859fcc13938" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 33325 } }
I20260927 04:02:15.291262 1760 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31ff61a46df749a7bd401859fcc13938 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31ff61a46df749a7bd401859fcc13938. Latest consensus state: current_term: 1 leader_uuid: "31ff61a46df749a7bd401859fcc13938" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31ff61a46df749a7bd401859fcc13938" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 33325 } } }
I20260927 04:02:15.291417 1760 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31ff61a46df749a7bd401859fcc13938 [sys.catalog]: This master's current role is: LEADER
I20260927 04:02:15.291386 1761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31ff61a46df749a7bd401859fcc13938 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31ff61a46df749a7bd401859fcc13938" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31ff61a46df749a7bd401859fcc13938" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 33325 } } }
I20260927 04:02:15.291504 1761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31ff61a46df749a7bd401859fcc13938 [sys.catalog]: This master's current role is: LEADER
I20260927 04:02:15.291783 1770 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:02:15.292265 1770 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:02:15.294287 1770 catalog_manager.cc:1347] Generated new cluster ID: 41055b53428f4d69b65e92ea2921e313
I20260927 04:02:15.294345 1770 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 04:02:15 dist-test-slave-glgz krb5kdc[1673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481735, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 04:02:15.299391 1759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:15.288266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59830 (local address 127.15.185.126:33325)
0927 04:02:15.288436 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:15.288444 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:15.288471 (+ 27us) server_negotiation.cc:407] Connection header received
0927 04:02:15.288661 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:15.288666 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:15.288721 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:15.288830 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:15.289638 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:15.290656 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:15.291803 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:15.292470 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:15.292964 (+ 494us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:15.295396 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:15.295401 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:15.295401 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:15.295413 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:15.297522 (+ 2109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:15.297892 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:15.297895 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:15.297896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:15.298029 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:15.298222 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:15.298223 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:15.298224 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:15.298551 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:15.298706 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:15.298796 (+ 90us) server_negotiation.cc:299] Negotiation successful
0927 04:02:15.298951 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":91,"threads_started":1}
I20260927 04:02:15.305938 1770 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:02:15.306665 1770 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:02:15.332851 1770 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31ff61a46df749a7bd401859fcc13938: Generated new TSK 0
I20260927 04:02:15.333093 1770 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.185.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab.
Entry for principal kudu/127.15.185.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab.
I20260927 04:02:15.362542 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.65:0
--local_ip_for_outbound_sockets=127.15.185.65
--webserver_interface=127.15.185.65
--webserver_port=0
--tserver_master_addrs=127.15.185.126:33325
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab
--principal=kudu/127.15.185.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:33103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:02:15.444660 1782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:02:15.444857 1782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:02:15.444880 1782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:02:15.446024 1782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:02:15.446067 1782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:02:15.446095 1782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.65
I20260927 04:02:15.447628 1782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:33103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab
--principal=kudu/127.15.185.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.185.65
--webserver_port=0
--tserver_master_addrs=127.15.185.126:33325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.65
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:02:15.447860 1782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:02:15.448093 1782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:02:15.453153 1790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:15.453158 1787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:15.453176 1788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:15.453198 1782 server_base.cc:1034] running on GCE node
I20260927 04:02:15.453585 1782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:02:15.453799 1782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:02:15.454939 1782 hybrid_clock.cc:648] HybridClock initialized: now 1790481735454926 us; error 28 us; skew 500 ppm
Sep 27 04:02:15 dist-test-slave-glgz krb5kdc[1673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481735, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:02:15.457472 1782 init.cc:377] Logged in from keytab as kudu/127.15.185.65@KRBTEST.COM (short username kudu)
I20260927 04:02:15.457964 1782 webserver.cc:460] Webserver started at http://127.15.185.65:37679/ using document root <none> and password file <none>
I20260927 04:02:15.458202 1782 fs_manager.cc:359] Metadata directory not provided
I20260927 04:02:15.458266 1782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:02:15.458412 1782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:02:15.459537 1782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/minicluster-data/ts-0/data/instance:
uuid: "c42271f2f5e1499cbdb699b3173eac7b"
format_stamp: "Formatted at 2026-09-27 04:02:15 on dist-test-slave-glgz"
server_key: "2239f5d03987e88e7962d0b2f192f03a"
server_key_iv: "402c5fb5498a32f177038efb53862a4c"
server_key_version: "encryptionkey@0"
I20260927 04:02:15.459903 1782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/minicluster-data/ts-0/wal/instance:
uuid: "c42271f2f5e1499cbdb699b3173eac7b"
format_stamp: "Formatted at 2026-09-27 04:02:15 on dist-test-slave-glgz"
server_key: "2239f5d03987e88e7962d0b2f192f03a"
server_key_iv: "402c5fb5498a32f177038efb53862a4c"
server_key_version: "encryptionkey@0"
I20260927 04:02:15.461179 1782 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 04:02:15.461922 1797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:15.462080 1782 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 04:02:15.462167 1782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/minicluster-data/ts-0/wal
uuid: "c42271f2f5e1499cbdb699b3173eac7b"
format_stamp: "Formatted at 2026-09-27 04:02:15 on dist-test-slave-glgz"
server_key: "2239f5d03987e88e7962d0b2f192f03a"
server_key_iv: "402c5fb5498a32f177038efb53862a4c"
server_key_version: "encryptionkey@0"
I20260927 04:02:15.462244 1782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:02:15.477821 1782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:02:15.480170 1782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:02:15.480335 1782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:02:15.480571 1782 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:02:15.480895 1782 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:02:15.480947 1782 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:15.480988 1782 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:02:15.481015 1782 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:15.486819 1782 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.65:39343
I20260927 04:02:15.486873 1911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.65:39343 every 8 connection(s)
I20260927 04:02:15.487265 1782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
I20260927 04:02:15.487772 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 1782
I20260927 04:02:15.487891 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/minicluster-data/ts-0/wal/instance
I20260927 04:02:15.488149 16101 external_mini_cluster.cc:1428] Setting key 0813dffa13adc2a45348fa98dbb8da10
Sep 27 04:02:15 dist-test-slave-glgz krb5kdc[1673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481735, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.65@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.185.66@KRBTEST.COM; defaulting to no policy
I20260927 04:02:15.497751 1759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:15.488032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:37027 (local address 127.15.185.126:33325)
0927 04:02:15.488117 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:15.488119 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:15.488325 (+ 206us) server_negotiation.cc:407] Connection header received
0927 04:02:15.488883 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:15.488884 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:15.488898 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:15.488972 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:15.489919 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:15.490503 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:15.491747 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:15.492136 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:15.492224 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:15.494665 (+ 2441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:15.494666 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:15.494667 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:15.494677 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:15.496449 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:15.496912 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:15.496913 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:15.496913 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:15.496966 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:15.497201 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:15.497202 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:15.497202 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:15.497305 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:15.497356 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:15.497607 (+ 251us) server_negotiation.cc:299] Negotiation successful
0927 04:02:15.497683 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 04:02:15.498014 1914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:15.488030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:33325 (local address 127.15.185.65:37027)
0927 04:02:15.488191 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:15.488233 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:15.488779 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:15.489027 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:15.489030 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:15.489608 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:02:15.489802 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:15.489807 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:15.490592 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:15.490593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:15.491634 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:15.491637 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:15.492258 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:15.492260 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:15.492391 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:15.492971 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:02:15.493130 (+ 159us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:02:15.494569 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:02:15.496563 (+ 1994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:15.496570 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:15.496573 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:15.496837 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:15.497025 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:15.497025 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:15.497026 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:15.497129 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:15.497357 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:02:15.497361 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:02:15.497504 (+ 143us) client_negotiation.cc:769] Sending connection context
0927 04:02:15.497586 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 04:02:15.497740 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":97,"threads_started":1}
I20260927 04:02:15.498607 1912 heartbeater.cc:347] Connected to a master server at 127.15.185.126:33325
I20260927 04:02:15.498729 1912 heartbeater.cc:464] Registering TS with master...
I20260927 04:02:15.498912 1912 heartbeater.cc:511] Master 127.15.185.126:33325 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.66@KRBTEST.COM" created.
I20260927 04:02:15.499547 1719 ts_manager.cc:194] Registered new tserver with Master: c42271f2f5e1499cbdb699b3173eac7b (127.15.185.65:39343)
I20260927 04:02:15.500173 1719 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.185.65@KRBTEST.COM'} at 127.15.185.65:37027
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab.
Entry for principal kudu/127.15.185.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab.
I20260927 04:02:15.515256 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.66:0
--local_ip_for_outbound_sockets=127.15.185.66
--webserver_interface=127.15.185.66
--webserver_port=0
--tserver_master_addrs=127.15.185.126:33325
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab
--principal=kudu/127.15.185.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:33103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:02:15.596319 1917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:02:15.596539 1917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:02:15.596565 1917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:02:15.597718 1917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:02:15.597760 1917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:02:15.597786 1917 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.66
I20260927 04:02:15.599256 1917 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:33103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab
--principal=kudu/127.15.185.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.185.66
--webserver_port=0
--tserver_master_addrs=127.15.185.126:33325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.66
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:02:15.599509 1917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:02:15.599741 1917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 04:02:15.605230 1917 server_base.cc:1034] running on GCE node
W20260927 04:02:15.605134 1925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:15.605137 1922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:15.605196 1923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:15.605590 1917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:02:15.605921 1917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:02:15.607137 1917 hybrid_clock.cc:648] HybridClock initialized: now 1790481735607115 us; error 50 us; skew 500 ppm
Sep 27 04:02:15 dist-test-slave-glgz krb5kdc[1673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481735, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:02:15.609853 1917 init.cc:377] Logged in from keytab as kudu/127.15.185.66@KRBTEST.COM (short username kudu)
I20260927 04:02:15.610309 1917 webserver.cc:460] Webserver started at http://127.15.185.66:39643/ using document root <none> and password file <none>
I20260927 04:02:15.610560 1917 fs_manager.cc:359] Metadata directory not provided
I20260927 04:02:15.610625 1917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:02:15.610795 1917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:02:15.611959 1917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/minicluster-data/ts-1/data/instance:
uuid: "d3f823d5222042968b0998118dc19186"
format_stamp: "Formatted at 2026-09-27 04:02:15 on dist-test-slave-glgz"
server_key: "caac5bf6c1eb8dbf4fd6a4830451efd2"
server_key_iv: "73ebd23d4ef9aa4f3512d2304ee6ffaf"
server_key_version: "encryptionkey@0"
I20260927 04:02:15.612325 1917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/minicluster-data/ts-1/wal/instance:
uuid: "d3f823d5222042968b0998118dc19186"
format_stamp: "Formatted at 2026-09-27 04:02:15 on dist-test-slave-glgz"
server_key: "caac5bf6c1eb8dbf4fd6a4830451efd2"
server_key_iv: "73ebd23d4ef9aa4f3512d2304ee6ffaf"
server_key_version: "encryptionkey@0"
I20260927 04:02:15.613765 1917 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 04:02:15.614511 1932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:15.614758 1917 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:02:15.614850 1917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/minicluster-data/ts-1/wal
uuid: "d3f823d5222042968b0998118dc19186"
format_stamp: "Formatted at 2026-09-27 04:02:15 on dist-test-slave-glgz"
server_key: "caac5bf6c1eb8dbf4fd6a4830451efd2"
server_key_iv: "73ebd23d4ef9aa4f3512d2304ee6ffaf"
server_key_version: "encryptionkey@0"
I20260927 04:02:15.614907 1917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:02:15.637108 1917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:02:15.638931 1917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:02:15.639079 1917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:02:15.639282 1917 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:02:15.639618 1917 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:02:15.639652 1917 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:15.639706 1917 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:02:15.639734 1917 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:15.645684 1917 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.66:38863
I20260927 04:02:15.645727 2046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.66:38863 every 8 connection(s)
I20260927 04:02:15.646081 1917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
I20260927 04:02:15.650347 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 1917
I20260927 04:02:15.650471 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/minicluster-data/ts-1/wal/instance
I20260927 04:02:15.650700 16101 external_mini_cluster.cc:1428] Setting key e08671dcebc1a79565fc8ea92e7bc5f8
Sep 27 04:02:15 dist-test-slave-glgz krb5kdc[1673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481735, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.66@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 04:02:15.656334 1759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:15.646838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:54743 (local address 127.15.185.126:33325)
0927 04:02:15.646912 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:15.646914 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:15.647166 (+ 252us) server_negotiation.cc:407] Connection header received
0927 04:02:15.647687 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:15.647689 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:15.647700 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:15.647761 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:15.648728 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:15.649247 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:15.650373 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:15.650628 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:15.650700 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:15.653351 (+ 2651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:15.653352 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:15.653353 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:15.653361 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:15.654900 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:15.655362 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:15.655364 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:15.655364 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:15.655423 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:15.655687 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:15.655687 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:15.655688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:15.655799 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:15.655871 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:15.656148 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 04:02:15.656238 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 04:02:15.656563 2049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:15.646819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:33325 (local address 127.15.185.66:54743)
0927 04:02:15.647011 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:15.647088 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:15.647567 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:15.647819 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:15.647822 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:15.648410 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:02:15.648615 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:15.648620 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:15.649353 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:15.649354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:15.650264 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:15.650267 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:15.650726 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:15.650727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:15.650888 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:15.651561 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:02:15.651572 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:02:15.653252 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:02:15.654977 (+ 1725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:15.654984 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:15.654987 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:15.655272 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:15.655494 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:15.655495 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:15.655495 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:15.655613 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:15.655861 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:02:15.655868 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:02:15.656016 (+ 148us) client_negotiation.cc:769] Sending connection context
0927 04:02:15.656117 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 04:02:15.656302 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":104,"threads_started":1}
I20260927 04:02:15.657131 2047 heartbeater.cc:347] Connected to a master server at 127.15.185.126:33325
I20260927 04:02:15.657255 2047 heartbeater.cc:464] Registering TS with master...
I20260927 04:02:15.657473 2047 heartbeater.cc:511] Master 127.15.185.126:33325 requested a full tablet report, sending...
I20260927 04:02:15.657902 1719 ts_manager.cc:194] Registered new tserver with Master: d3f823d5222042968b0998118dc19186 (127.15.185.66:38863)
I20260927 04:02:15.658344 1719 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.185.66@KRBTEST.COM'} at 127.15.185.66:54743
WARNING: no policy specified for kudu/127.15.185.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab.
Entry for principal kudu/127.15.185.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab.
I20260927 04:02:15.677500 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.67:0
--local_ip_for_outbound_sockets=127.15.185.67
--webserver_interface=127.15.185.67
--webserver_port=0
--tserver_master_addrs=127.15.185.126:33325
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab
--principal=kudu/127.15.185.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:33103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:02:15.760004 2052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:02:15.760224 2052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:02:15.760247 2052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:02:15.761422 2052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:02:15.761466 2052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:02:15.761492 2052 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.67
I20260927 04:02:15.762981 2052 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:33103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab
--principal=kudu/127.15.185.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.185.67
--webserver_port=0
--tserver_master_addrs=127.15.185.126:33325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2052
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.67
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:02:15.763273 2052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:02:15.763484 2052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:02:15.768445 2058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:15.768443 2060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:15.768448 2057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:15.768781 2052 server_base.cc:1034] running on GCE node
I20260927 04:02:15.769013 2052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:02:15.769229 2052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:02:15.770371 2052 hybrid_clock.cc:648] HybridClock initialized: now 1790481735770359 us; error 29 us; skew 500 ppm
Sep 27 04:02:15 dist-test-slave-glgz krb5kdc[1673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481735, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:02:15.772814 2052 init.cc:377] Logged in from keytab as kudu/127.15.185.67@KRBTEST.COM (short username kudu)
I20260927 04:02:15.773213 2052 webserver.cc:460] Webserver started at http://127.15.185.67:40991/ using document root <none> and password file <none>
I20260927 04:02:15.773402 2052 fs_manager.cc:359] Metadata directory not provided
I20260927 04:02:15.773442 2052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:02:15.773530 2052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:02:15.774541 2052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/minicluster-data/ts-2/data/instance:
uuid: "23689d8455cb4f40b9a2b62d2a9e8c80"
format_stamp: "Formatted at 2026-09-27 04:02:15 on dist-test-slave-glgz"
server_key: "53b18a39efc2acae3fb85ae0fafb220d"
server_key_iv: "63aa9b37e641a05625d0b17ddaac7eef"
server_key_version: "encryptionkey@0"
I20260927 04:02:15.774828 2052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/minicluster-data/ts-2/wal/instance:
uuid: "23689d8455cb4f40b9a2b62d2a9e8c80"
format_stamp: "Formatted at 2026-09-27 04:02:15 on dist-test-slave-glgz"
server_key: "53b18a39efc2acae3fb85ae0fafb220d"
server_key_iv: "63aa9b37e641a05625d0b17ddaac7eef"
server_key_version: "encryptionkey@0"
I20260927 04:02:15.776110 2052 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 04:02:15.776762 2067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:15.776908 2052 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 04:02:15.776986 2052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/minicluster-data/ts-2/wal
uuid: "23689d8455cb4f40b9a2b62d2a9e8c80"
format_stamp: "Formatted at 2026-09-27 04:02:15 on dist-test-slave-glgz"
server_key: "53b18a39efc2acae3fb85ae0fafb220d"
server_key_iv: "63aa9b37e641a05625d0b17ddaac7eef"
server_key_version: "encryptionkey@0"
I20260927 04:02:15.777041 2052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:02:15.784157 2052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:02:15.786079 2052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:02:15.786204 2052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:02:15.786417 2052 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:02:15.786720 2052 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:02:15.786758 2052 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:15.786782 2052 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:02:15.786794 2052 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:15.792827 2052 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.67:37007
I20260927 04:02:15.792902 2181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.67:37007 every 8 connection(s)
I20260927 04:02:15.793243 2052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
Sep 27 04:02:15 dist-test-slave-glgz krb5kdc[1673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481735, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.67@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 04:02:15.802508 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 2052
I20260927 04:02:15.802628 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/minicluster-data/ts-2/wal/instance
I20260927 04:02:15.802836 16101 external_mini_cluster.cc:1428] Setting key 799ba013c5e88684159270cad0d10827
I20260927 04:02:15.803001 1759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:15.793982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:58527 (local address 127.15.185.126:33325)
0927 04:02:15.794026 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:15.794027 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:15.794269 (+ 242us) server_negotiation.cc:407] Connection header received
0927 04:02:15.794798 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:15.794799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:15.794812 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:15.794864 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:15.795908 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:15.796406 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:15.797413 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:15.797602 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:15.797684 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:15.799896 (+ 2212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:15.799898 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:15.799899 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:15.799906 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:15.801400 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:15.801808 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:15.801809 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:15.801810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:15.801870 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:15.802116 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:15.802117 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:15.802117 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:15.802239 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:15.802313 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:15.802784 (+ 471us) server_negotiation.cc:299] Negotiation successful
0927 04:02:15.802895 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 04:02:15.803095 2184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:15.793974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:33325 (local address 127.15.185.67:58527)
0927 04:02:15.794143 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:15.794181 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:15.794685 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:15.794951 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:15.794955 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:15.795605 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:02:15.795810 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:15.795814 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:15.796498 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:15.796499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:15.797309 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:15.797311 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:15.797703 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:15.797704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:15.797831 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:15.798360 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:02:15.798372 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:02:15.799784 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:02:15.801480 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:15.801487 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:15.801489 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:15.801721 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:15.801938 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:15.801939 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:15.801939 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:15.802050 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:15.802303 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:02:15.802309 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:02:15.802487 (+ 178us) client_negotiation.cc:769] Sending connection context
0927 04:02:15.802618 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 04:02:15.802799 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":77,"threads_started":1}
I20260927 04:02:15.803537 2182 heartbeater.cc:347] Connected to a master server at 127.15.185.126:33325
I20260927 04:02:15.803651 2182 heartbeater.cc:464] Registering TS with master...
I20260927 04:02:15.803861 2182 heartbeater.cc:511] Master 127.15.185.126:33325 requested a full tablet report, sending...
I20260927 04:02:15.804206 1719 ts_manager.cc:194] Registered new tserver with Master: 23689d8455cb4f40b9a2b62d2a9e8c80 (127.15.185.67:37007)
I20260927 04:02:15.804579 1719 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.185.67@KRBTEST.COM'} at 127.15.185.67:58527
I20260927 04:02:15.815073 16101 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 04:02:15.822993 1759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:15.816515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59832 (local address 127.15.185.126:33325)
0927 04:02:15.816576 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:15.816578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:15.816693 (+ 115us) server_negotiation.cc:407] Connection header received
0927 04:02:15.816771 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:15.816772 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:15.816783 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:15.816860 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:15.817476 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:15.818086 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:15.818890 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:15.819142 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:15.819219 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:15.820538 (+ 1319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:15.820539 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:15.820540 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:15.820547 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:15.821729 (+ 1182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:15.822122 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:15.822123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:15.822124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:15.822197 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:15.822410 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:15.822411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:15.822412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:15.822557 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:15.822667 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:15.822824 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 04:02:15.822891 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 04:02:15.824692 1719 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59832:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 04:02:15.826344 1719 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 04:02:15.833076 2192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:15.828504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58736 (local address 127.15.185.67:37007)
0927 04:02:15.828817 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:15.828821 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:15.828831 (+ 10us) server_negotiation.cc:407] Connection header received
0927 04:02:15.828870 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:15.828872 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:15.828885 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:15.828986 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:02:15.829175 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:15.830330 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:15.831723 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:15.832515 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:15.832590 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:15.832652 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:15.832942 (+ 290us) server_negotiation.cc:299] Negotiation successful
0927 04:02:15.832987 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":82,"threads_started":1}
I20260927 04:02:15.833082 2190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:15.828451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.65:39343 (local address 127.0.0.1:39882)
0927 04:02:15.828672 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:15.828684 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:15.828735 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:15.829439 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:15.829440 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:15.829443 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:02:15.829497 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:15.829498 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:15.830413 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:15.830414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:15.831772 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:15.831775 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:15.832705 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:15.832706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:15.832807 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:15.832814 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 04:02:15.832900 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 04:02:15.832980 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"mutex_wait_us":60,"thread_start_us":68,"threads_started":1}
I20260927 04:02:15.833082 2191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:15.828422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.67:37007 (local address 127.0.0.1:58736)
0927 04:02:15.828589 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:15.828637 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:15.828702 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:15.829010 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:15.829013 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:15.829015 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:02:15.829102 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:15.829104 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:15.830446 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:15.830448 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:15.831623 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:15.831625 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:15.832729 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:15.832730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:15.832798 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:15.832814 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 04:02:15.832900 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 04:02:15.832980 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":73,"thread_start_us":54,"threads_started":1}
I20260927 04:02:15.833684 2194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:15.828625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.66:38863 (local address 127.0.0.1:39114)
0927 04:02:15.828950 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:15.828979 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:15.829035 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:15.829290 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:15.829291 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:15.829293 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:02:15.829339 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:15.829340 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:15.830527 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:15.830528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:15.831773 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:15.831775 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:15.833484 (+ 1709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:15.833485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:15.833559 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:15.833563 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 04:02:15.833607 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 04:02:15.833636 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":108,"thread_start_us":48,"threads_started":1}
I20260927 04:02:15.833755 2195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:15.828814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39114 (local address 127.15.185.66:38863)
0927 04:02:15.828985 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:15.828988 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:15.829025 (+ 37us) server_negotiation.cc:407] Connection header received
0927 04:02:15.829095 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:15.829097 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:15.829110 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:15.829206 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:02:15.829414 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:15.830439 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:15.832699 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:15.833401 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:15.833474 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:15.833542 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:15.833620 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 04:02:15.833660 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":77,"threads_started":1}
I20260927 04:02:15.834149 2193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:15.828560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39882 (local address 127.15.185.65:39343)
0927 04:02:15.829177 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:15.829182 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:15.829191 (+ 9us) server_negotiation.cc:407] Connection header received
0927 04:02:15.829228 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:15.829230 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:15.829245 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:15.829321 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:02:15.829602 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:15.830311 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:15.831853 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:15.832507 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:15.832590 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:15.832652 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:15.834001 (+ 1349us) server_negotiation.cc:299] Negotiation successful
0927 04:02:15.834074 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":97,"threads_started":1}
I20260927 04:02:15.834426 2115 tablet_service.cc:1467] Processing CreateTablet for tablet b092db6c71a54e219e9d8e6dea30e426 (DEFAULT_TABLE table=test-table [id=e815303588e746f6af8c2cbefc0b5887]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:02:15.834636 1980 tablet_service.cc:1467] Processing CreateTablet for tablet b092db6c71a54e219e9d8e6dea30e426 (DEFAULT_TABLE table=test-table [id=e815303588e746f6af8c2cbefc0b5887]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:02:15.834781 2115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b092db6c71a54e219e9d8e6dea30e426. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:02:15.834961 1980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b092db6c71a54e219e9d8e6dea30e426. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:02:15.837520 2196 tablet_bootstrap.cc:492] T b092db6c71a54e219e9d8e6dea30e426 P 23689d8455cb4f40b9a2b62d2a9e8c80: Bootstrap starting.
I20260927 04:02:15.837632 2197 tablet_bootstrap.cc:492] T b092db6c71a54e219e9d8e6dea30e426 P d3f823d5222042968b0998118dc19186: Bootstrap starting.
I20260927 04:02:15.838244 2196 tablet_bootstrap.cc:654] T b092db6c71a54e219e9d8e6dea30e426 P 23689d8455cb4f40b9a2b62d2a9e8c80: Neither blocks nor log segments found. Creating new log.
I20260927 04:02:15.838347 2197 tablet_bootstrap.cc:654] T b092db6c71a54e219e9d8e6dea30e426 P d3f823d5222042968b0998118dc19186: Neither blocks nor log segments found. Creating new log.
I20260927 04:02:15.838397 1845 tablet_service.cc:1467] Processing CreateTablet for tablet b092db6c71a54e219e9d8e6dea30e426 (DEFAULT_TABLE table=test-table [id=e815303588e746f6af8c2cbefc0b5887]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:02:15.838783 1845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b092db6c71a54e219e9d8e6dea30e426. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:02:15.838814 2196 log.cc:824] T b092db6c71a54e219e9d8e6dea30e426 P 23689d8455cb4f40b9a2b62d2a9e8c80: Log is configured to *not* fsync() on all Append() calls
I20260927 04:02:15.838945 2197 log.cc:824] T b092db6c71a54e219e9d8e6dea30e426 P d3f823d5222042968b0998118dc19186: Log is configured to *not* fsync() on all Append() calls
I20260927 04:02:15.839565 2196 tablet_bootstrap.cc:492] T b092db6c71a54e219e9d8e6dea30e426 P 23689d8455cb4f40b9a2b62d2a9e8c80: No bootstrap required, opened a new log
I20260927 04:02:15.839622 2196 ts_tablet_manager.cc:1397] T b092db6c71a54e219e9d8e6dea30e426 P 23689d8455cb4f40b9a2b62d2a9e8c80: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 04:02:15.841209 2196 raft_consensus.cc:348] T b092db6c71a54e219e9d8e6dea30e426 P 23689d8455cb4f40b9a2b62d2a9e8c80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23689d8455cb4f40b9a2b62d2a9e8c80" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 37007 } } peers { permanent_uuid: "d3f823d5222042968b0998118dc19186" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38863 } } peers { permanent_uuid: "c42271f2f5e1499cbdb699b3173eac7b" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 39343 } }
I20260927 04:02:15.841354 2196 raft_consensus.cc:374] T b092db6c71a54e219e9d8e6dea30e426 P 23689d8455cb4f40b9a2b62d2a9e8c80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:02:15.841377 2196 raft_consensus.cc:729] T b092db6c71a54e219e9d8e6dea30e426 P 23689d8455cb4f40b9a2b62d2a9e8c80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23689d8455cb4f40b9a2b62d2a9e8c80, State: Initialized, Role: FOLLOWER
I20260927 04:02:15.841498 2196 consensus_queue.cc:260] T b092db6c71a54e219e9d8e6dea30e426 P 23689d8455cb4f40b9a2b62d2a9e8c80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23689d8455cb4f40b9a2b62d2a9e8c80" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 37007 } } peers { permanent_uuid: "d3f823d5222042968b0998118dc19186" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38863 } } peers { permanent_uuid: "c42271f2f5e1499cbdb699b3173eac7b" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 39343 } }
I20260927 04:02:15.841645 2200 tablet_bootstrap.cc:492] T b092db6c71a54e219e9d8e6dea30e426 P c42271f2f5e1499cbdb699b3173eac7b: Bootstrap starting.
I20260927 04:02:15.841794 2196 ts_tablet_manager.cc:1428] T b092db6c71a54e219e9d8e6dea30e426 P 23689d8455cb4f40b9a2b62d2a9e8c80: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 04:02:15.841918 2182 heartbeater.cc:503] Master 127.15.185.126:33325 was elected leader, sending a full tablet report...
I20260927 04:02:15.842345 2200 tablet_bootstrap.cc:654] T b092db6c71a54e219e9d8e6dea30e426 P c42271f2f5e1499cbdb699b3173eac7b: Neither blocks nor log segments found. Creating new log.
I20260927 04:02:15.842417 2197 tablet_bootstrap.cc:492] T b092db6c71a54e219e9d8e6dea30e426 P d3f823d5222042968b0998118dc19186: No bootstrap required, opened a new log
I20260927 04:02:15.842489 2197 ts_tablet_manager.cc:1397] T b092db6c71a54e219e9d8e6dea30e426 P d3f823d5222042968b0998118dc19186: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260927 04:02:15.842912 2200 log.cc:824] T b092db6c71a54e219e9d8e6dea30e426 P c42271f2f5e1499cbdb699b3173eac7b: Log is configured to *not* fsync() on all Append() calls
I20260927 04:02:15.843652 2200 tablet_bootstrap.cc:492] T b092db6c71a54e219e9d8e6dea30e426 P c42271f2f5e1499cbdb699b3173eac7b: No bootstrap required, opened a new log
I20260927 04:02:15.843731 2200 ts_tablet_manager.cc:1397] T b092db6c71a54e219e9d8e6dea30e426 P c42271f2f5e1499cbdb699b3173eac7b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 04:02:15.844262 2197 raft_consensus.cc:348] T b092db6c71a54e219e9d8e6dea30e426 P d3f823d5222042968b0998118dc19186 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23689d8455cb4f40b9a2b62d2a9e8c80" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 37007 } } peers { permanent_uuid: "d3f823d5222042968b0998118dc19186" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38863 } } peers { permanent_uuid: "c42271f2f5e1499cbdb699b3173eac7b" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 39343 } }
I20260927 04:02:15.844398 2197 raft_consensus.cc:374] T b092db6c71a54e219e9d8e6dea30e426 P d3f823d5222042968b0998118dc19186 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:02:15.844420 2197 raft_consensus.cc:729] T b092db6c71a54e219e9d8e6dea30e426 P d3f823d5222042968b0998118dc19186 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3f823d5222042968b0998118dc19186, State: Initialized, Role: FOLLOWER
I20260927 04:02:15.844549 2197 consensus_queue.cc:260] T b092db6c71a54e219e9d8e6dea30e426 P d3f823d5222042968b0998118dc19186 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23689d8455cb4f40b9a2b62d2a9e8c80" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 37007 } } peers { permanent_uuid: "d3f823d5222042968b0998118dc19186" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38863 } } peers { permanent_uuid: "c42271f2f5e1499cbdb699b3173eac7b" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 39343 } }
I20260927 04:02:15.844834 2047 heartbeater.cc:503] Master 127.15.185.126:33325 was elected leader, sending a full tablet report...
I20260927 04:02:15.844834 2197 ts_tablet_manager.cc:1428] T b092db6c71a54e219e9d8e6dea30e426 P d3f823d5222042968b0998118dc19186: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 04:02:15.845580 2200 raft_consensus.cc:348] T b092db6c71a54e219e9d8e6dea30e426 P c42271f2f5e1499cbdb699b3173eac7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23689d8455cb4f40b9a2b62d2a9e8c80" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 37007 } } peers { permanent_uuid: "d3f823d5222042968b0998118dc19186" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38863 } } peers { permanent_uuid: "c42271f2f5e1499cbdb699b3173eac7b" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 39343 } }
I20260927 04:02:15.845729 2200 raft_consensus.cc:374] T b092db6c71a54e219e9d8e6dea30e426 P c42271f2f5e1499cbdb699b3173eac7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:02:15.845773 2200 raft_consensus.cc:729] T b092db6c71a54e219e9d8e6dea30e426 P c42271f2f5e1499cbdb699b3173eac7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c42271f2f5e1499cbdb699b3173eac7b, State: Initialized, Role: FOLLOWER
I20260927 04:02:15.845882 2200 consensus_queue.cc:260] T b092db6c71a54e219e9d8e6dea30e426 P c42271f2f5e1499cbdb699b3173eac7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23689d8455cb4f40b9a2b62d2a9e8c80" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 37007 } } peers { permanent_uuid: "d3f823d5222042968b0998118dc19186" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38863 } } peers { permanent_uuid: "c42271f2f5e1499cbdb699b3173eac7b" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 39343 } }
I20260927 04:02:15.846114 2200 ts_tablet_manager.cc:1428] T b092db6c71a54e219e9d8e6dea30e426 P c42271f2f5e1499cbdb699b3173eac7b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 04:02:15.846163 1912 heartbeater.cc:503] Master 127.15.185.126:33325 was elected leader, sending a full tablet report...
I20260927 04:02:15.884428 2204 raft_consensus.cc:482] T b092db6c71a54e219e9d8e6dea30e426 P c42271f2f5e1499cbdb699b3173eac7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:02:15.884564 2204 raft_consensus.cc:504] T b092db6c71a54e219e9d8e6dea30e426 P c42271f2f5e1499cbdb699b3173eac7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23689d8455cb4f40b9a2b62d2a9e8c80" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 37007 } } peers { permanent_uuid: "d3f823d5222042968b0998118dc19186" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38863 } } peers { permanent_uuid: "c42271f2f5e1499cbdb699b3173eac7b" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 39343 } }
I20260927 04:02:15.884907 2204 leader_election.cc:288] T b092db6c71a54e219e9d8e6dea30e426 P c42271f2f5e1499cbdb699b3173eac7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23689d8455cb4f40b9a2b62d2a9e8c80 (127.15.185.67:37007), d3f823d5222042968b0998118dc19186 (127.15.185.66:38863)
I20260927 04:02:15.888936 2195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:15.885321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:44345 (local address 127.15.185.66:38863)
0927 04:02:15.885382 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:15.885384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:15.885601 (+ 217us) server_negotiation.cc:407] Connection header received
0927 04:02:15.885638 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:15.885639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:15.885653 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:15.885730 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:02:15.885832 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:15.886540 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:15.887941 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:15.888534 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:15.888620 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:15.888628 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:15.888818 (+ 190us) server_negotiation.cc:299] Negotiation successful
0927 04:02:15.888862 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 04:02:15.888936 2192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:15.885224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:51569 (local address 127.15.185.67:37007)
0927 04:02:15.885275 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:15.885278 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:15.885288 (+ 10us) server_negotiation.cc:407] Connection header received
0927 04:02:15.885318 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:15.885319 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:15.885332 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:15.885427 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:02:15.885670 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:15.886434 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:15.887941 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:15.888534 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:15.888622 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:15.888629 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:15.888824 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 04:02:15.888862 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 04:02:15.888937 1914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:15.885098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.67:37007 (local address 127.15.185.65:51569)
0927 04:02:15.885150 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:15.885163 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:15.885211 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:15.885442 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:15.885443 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:15.885445 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:02:15.885524 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:15.885526 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:15.886512 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:15.886512 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:15.887851 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:15.887854 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:15.888625 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:15.888626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:15.888752 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:15.888755 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 04:02:15.888807 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 04:02:15.888836 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":26}
I20260927 04:02:15.888943 2205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:15.885268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.66:38863 (local address 127.15.185.65:44345)
0927 04:02:15.885490 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:15.885501 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:15.885560 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:15.885739 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:15.885739 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:15.885741 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:02:15.885787 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:15.885788 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:15.886608 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:15.886609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:15.887851 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:15.887854 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:15.888625 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:15.888626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:15.888710 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:15.888714 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 04:02:15.888779 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 04:02:15.888830 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":32,"thread_start_us":151,"threads_started":1}
I20260927 04:02:15.889398 2000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b092db6c71a54e219e9d8e6dea30e426" candidate_uuid: "c42271f2f5e1499cbdb699b3173eac7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3f823d5222042968b0998118dc19186" is_pre_election: true
I20260927 04:02:15.889396 2135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b092db6c71a54e219e9d8e6dea30e426" candidate_uuid: "c42271f2f5e1499cbdb699b3173eac7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23689d8455cb4f40b9a2b62d2a9e8c80" is_pre_election: true
I20260927 04:02:15.889570 2135 raft_consensus.cc:2454] T b092db6c71a54e219e9d8e6dea30e426 P 23689d8455cb4f40b9a2b62d2a9e8c80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c42271f2f5e1499cbdb699b3173eac7b in term 0.
I20260927 04:02:15.889570 2000 raft_consensus.cc:2454] T b092db6c71a54e219e9d8e6dea30e426 P d3f823d5222042968b0998118dc19186 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c42271f2f5e1499cbdb699b3173eac7b in term 0.
I20260927 04:02:15.889825 1799 leader_election.cc:302] T b092db6c71a54e219e9d8e6dea30e426 P c42271f2f5e1499cbdb699b3173eac7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23689d8455cb4f40b9a2b62d2a9e8c80, c42271f2f5e1499cbdb699b3173eac7b; no voters:
I20260927 04:02:15.889961 2204 raft_consensus.cc:2781] T b092db6c71a54e219e9d8e6dea30e426 P c42271f2f5e1499cbdb699b3173eac7b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 04:02:15.890020 2204 raft_consensus.cc:482] T b092db6c71a54e219e9d8e6dea30e426 P c42271f2f5e1499cbdb699b3173eac7b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 04:02:15.890045 2204 raft_consensus.cc:3037] T b092db6c71a54e219e9d8e6dea30e426 P c42271f2f5e1499cbdb699b3173eac7b [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:02:15.890838 2204 raft_consensus.cc:504] T b092db6c71a54e219e9d8e6dea30e426 P c42271f2f5e1499cbdb699b3173eac7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23689d8455cb4f40b9a2b62d2a9e8c80" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 37007 } } peers { permanent_uuid: "d3f823d5222042968b0998118dc19186" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38863 } } peers { permanent_uuid: "c42271f2f5e1499cbdb699b3173eac7b" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 39343 } }
I20260927 04:02:15.890973 2204 leader_election.cc:288] T b092db6c71a54e219e9d8e6dea30e426 P c42271f2f5e1499cbdb699b3173eac7b [CANDIDATE]: Term 1 election: Requested vote from peers 23689d8455cb4f40b9a2b62d2a9e8c80 (127.15.185.67:37007), d3f823d5222042968b0998118dc19186 (127.15.185.66:38863)
I20260927 04:02:15.891151 2135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b092db6c71a54e219e9d8e6dea30e426" candidate_uuid: "c42271f2f5e1499cbdb699b3173eac7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23689d8455cb4f40b9a2b62d2a9e8c80"
I20260927 04:02:15.891161 2000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b092db6c71a54e219e9d8e6dea30e426" candidate_uuid: "c42271f2f5e1499cbdb699b3173eac7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3f823d5222042968b0998118dc19186"
I20260927 04:02:15.891220 2135 raft_consensus.cc:3037] T b092db6c71a54e219e9d8e6dea30e426 P 23689d8455cb4f40b9a2b62d2a9e8c80 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:02:15.891258 2000 raft_consensus.cc:3037] T b092db6c71a54e219e9d8e6dea30e426 P d3f823d5222042968b0998118dc19186 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:02:15.892082 2135 raft_consensus.cc:2454] T b092db6c71a54e219e9d8e6dea30e426 P 23689d8455cb4f40b9a2b62d2a9e8c80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c42271f2f5e1499cbdb699b3173eac7b in term 1.
I20260927 04:02:15.892197 2000 raft_consensus.cc:2454] T b092db6c71a54e219e9d8e6dea30e426 P d3f823d5222042968b0998118dc19186 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c42271f2f5e1499cbdb699b3173eac7b in term 1.
I20260927 04:02:15.892257 1799 leader_election.cc:302] T b092db6c71a54e219e9d8e6dea30e426 P c42271f2f5e1499cbdb699b3173eac7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23689d8455cb4f40b9a2b62d2a9e8c80, c42271f2f5e1499cbdb699b3173eac7b; no voters:
I20260927 04:02:15.892346 2204 raft_consensus.cc:2781] T b092db6c71a54e219e9d8e6dea30e426 P c42271f2f5e1499cbdb699b3173eac7b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:02:15.892522 2204 raft_consensus.cc:686] T b092db6c71a54e219e9d8e6dea30e426 P c42271f2f5e1499cbdb699b3173eac7b [term 1 LEADER]: Becoming Leader. State: Replica: c42271f2f5e1499cbdb699b3173eac7b, State: Running, Role: LEADER
I20260927 04:02:15.892629 2204 consensus_queue.cc:237] T b092db6c71a54e219e9d8e6dea30e426 P c42271f2f5e1499cbdb699b3173eac7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23689d8455cb4f40b9a2b62d2a9e8c80" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 37007 } } peers { permanent_uuid: "d3f823d5222042968b0998118dc19186" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38863 } } peers { permanent_uuid: "c42271f2f5e1499cbdb699b3173eac7b" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 39343 } }
I20260927 04:02:15.893464 1719 catalog_manager.cc:5495] T b092db6c71a54e219e9d8e6dea30e426 P c42271f2f5e1499cbdb699b3173eac7b reported cstate change: term changed from 0 to 1, leader changed from <none> to c42271f2f5e1499cbdb699b3173eac7b (127.15.185.65). New cstate: current_term: 1 leader_uuid: "c42271f2f5e1499cbdb699b3173eac7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23689d8455cb4f40b9a2b62d2a9e8c80" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 37007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3f823d5222042968b0998118dc19186" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c42271f2f5e1499cbdb699b3173eac7b" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 39343 } health_report { overall_health: HEALTHY } } }
W20260927 04:02:15.896557 2048 tablet.cc:2369] T b092db6c71a54e219e9d8e6dea30e426 P d3f823d5222042968b0998118dc19186: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 04:02:15.906435 2193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:15.903948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39884 (local address 127.15.185.65:39343)
0927 04:02:15.904011 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:15.904013 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:15.904024 (+ 11us) server_negotiation.cc:407] Connection header received
0927 04:02:15.904100 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:15.904101 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:15.904124 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:15.904208 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:02:15.904358 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:15.904871 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:15.905630 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:15.905834 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:15.905903 (+ 69us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:15.906068 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:02:15.906166 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:02:15.906287 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:15.906317 (+ 30us) server_negotiation.cc:299] Negotiation successful
0927 04:02:15.906331 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 04:02:15.908371 2000 raft_consensus.cc:1261] T b092db6c71a54e219e9d8e6dea30e426 P d3f823d5222042968b0998118dc19186 [term 1 FOLLOWER]: Refusing update from remote peer c42271f2f5e1499cbdb699b3173eac7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:02:15.908420 2135 raft_consensus.cc:1261] T b092db6c71a54e219e9d8e6dea30e426 P 23689d8455cb4f40b9a2b62d2a9e8c80 [term 1 FOLLOWER]: Refusing update from remote peer c42271f2f5e1499cbdb699b3173eac7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:02:15.908763 2204 consensus_queue.cc:1035] T b092db6c71a54e219e9d8e6dea30e426 P c42271f2f5e1499cbdb699b3173eac7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3f823d5222042968b0998118dc19186" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 38863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:02:15.908929 2206 consensus_queue.cc:1035] T b092db6c71a54e219e9d8e6dea30e426 P c42271f2f5e1499cbdb699b3173eac7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "23689d8455cb4f40b9a2b62d2a9e8c80" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 37007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:02:15.911293 2209 mvcc.cc:205] Tried to move back new op lower bound from 7333813190277529600 to 7333813190216847360. Current Snapshot: MvccSnapshot[applied={T|T < 7333813190277529600}]
I20260927 04:02:15.911391 2211 mvcc.cc:205] Tried to move back new op lower bound from 7333813190277529600 to 7333813190216847360. Current Snapshot: MvccSnapshot[applied={T|T < 7333813190277529600}]
I20260927 04:02:15.913690 1718 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:02:15.913789 1718 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:02:15.914685 1718 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 31ff61a46df749a7bd401859fcc13938: Sending DeleteTablet for 3 replicas of tablet b092db6c71a54e219e9d8e6dea30e426
I20260927 04:02:15.914994 1845 tablet_service.cc:1514] Processing DeleteTablet for tablet b092db6c71a54e219e9d8e6dea30e426 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:02:15 UTC) from {username='kudu', principal='kudu/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:39882
I20260927 04:02:15.915094 1980 tablet_service.cc:1514] Processing DeleteTablet for tablet b092db6c71a54e219e9d8e6dea30e426 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:02:15 UTC) from {username='kudu', principal='kudu/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:39114
I20260927 04:02:15.915269 2115 tablet_service.cc:1514] Processing DeleteTablet for tablet b092db6c71a54e219e9d8e6dea30e426 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:02:15 UTC) from {username='kudu', principal='kudu/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:58736
I20260927 04:02:15.915337 2220 tablet_replica.cc:330] T b092db6c71a54e219e9d8e6dea30e426 P d3f823d5222042968b0998118dc19186: stopping tablet replica
I20260927 04:02:15.915438 2220 raft_consensus.cc:2229] T b092db6c71a54e219e9d8e6dea30e426 P d3f823d5222042968b0998118dc19186 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:02:15.915481 2221 tablet_replica.cc:330] T b092db6c71a54e219e9d8e6dea30e426 P 23689d8455cb4f40b9a2b62d2a9e8c80: stopping tablet replica
I20260927 04:02:15.915596 2220 raft_consensus.cc:2258] T b092db6c71a54e219e9d8e6dea30e426 P d3f823d5222042968b0998118dc19186 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:02:15.915602 2221 raft_consensus.cc:2229] T b092db6c71a54e219e9d8e6dea30e426 P 23689d8455cb4f40b9a2b62d2a9e8c80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:02:15.915699 2221 raft_consensus.cc:2258] T b092db6c71a54e219e9d8e6dea30e426 P 23689d8455cb4f40b9a2b62d2a9e8c80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:02:15.915782 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 1782
I20260927 04:02:15.916826 2221 ts_tablet_manager.cc:1905] T b092db6c71a54e219e9d8e6dea30e426 P 23689d8455cb4f40b9a2b62d2a9e8c80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:02:15.918723 2221 ts_tablet_manager.cc:1918] T b092db6c71a54e219e9d8e6dea30e426 P 23689d8455cb4f40b9a2b62d2a9e8c80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:02:15.918789 2221 log.cc:1196] T b092db6c71a54e219e9d8e6dea30e426 P 23689d8455cb4f40b9a2b62d2a9e8c80: Deleting WAL directory at /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481498018407-16101-0/minicluster-data/ts-2/wal/wals/b092db6c71a54e219e9d8e6dea30e426
I20260927 04:02:15.919082 2221 ts_tablet_manager.cc:1939] T b092db6c71a54e219e9d8e6dea30e426 P 23689d8455cb4f40b9a2b62d2a9e8c80: Deleting consensus metadata
I20260927 04:02:15.919515 1704 catalog_manager.cc:4841] TS 23689d8455cb4f40b9a2b62d2a9e8c80 (127.15.185.67:37007): tablet b092db6c71a54e219e9d8e6dea30e426 (table test-table [id=e815303588e746f6af8c2cbefc0b5887]) successfully deleted
W20260927 04:02:15.921474 1706 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.185.65:39343 (error 108)
I20260927 04:02:15.921731 2220 ts_tablet_manager.cc:1905] T b092db6c71a54e219e9d8e6dea30e426 P d3f823d5222042968b0998118dc19186: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:02:15.921773 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 1917
I20260927 04:02:15.922689 2194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:15.922550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.65:39343 (local address 127.0.0.1:39888)
0927 04:02:15.922616 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:15.922653 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.65:39343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 04:02:15.922827 1706 catalog_manager.cc:4584] TS c42271f2f5e1499cbdb699b3173eac7b (127.15.185.65:39343): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b092db6c71a54e219e9d8e6dea30e426: Network error: Client connection negotiation failed: client connection to 127.15.185.65:39343: connect: Connection refused (error 111)
I20260927 04:02:15.926752 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 2052
I20260927 04:02:15.926916 2194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:15.926782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.66:38863 (local address 127.0.0.1:39120)
0927 04:02:15.926844 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:15.926877 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.66:38863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 04:02:15.931777 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 1687
2026-09-27T04:02:15Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2001 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-27T04:02:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:02:15Z Disabled control of system clock
I20260927 04:02:15.959625 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:36163
--webserver_interface=127.15.185.126
--webserver_port=0
--builtin_ntp_servers=127.15.185.84:43055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.185.126:36163
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 04:02:16.041450 2226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:02:16.041692 2226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:02:16.041713 2226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:02:16.042857 2226 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:02:16.042896 2226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:02:16.042909 2226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:02:16.042922 2226 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:02:16.042934 2226 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:02:16.044390 2226 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:43055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.185.126:36163
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:36163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2226
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:02:16.044612 2226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:02:16.044874 2226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:02:16.049873 2234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:16.049881 2231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:16.049885 2232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:16.049947 2226 server_base.cc:1034] running on GCE node
I20260927 04:02:16.050343 2226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:02:16.050709 2226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:02:16.051865 2226 hybrid_clock.cc:648] HybridClock initialized: now 1790481736051838 us; error 44 us; skew 500 ppm
I20260927 04:02:16.053100 2226 webserver.cc:460] Webserver started at http://127.15.185.126:39273/ using document root <none> and password file <none>
I20260927 04:02:16.053363 2226 fs_manager.cc:359] Metadata directory not provided
I20260927 04:02:16.053411 2226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:02:16.053622 2226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:02:16.054759 2226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/master-0/data/instance:
uuid: "e32be32ca382450a9339506e33c983a2"
format_stamp: "Formatted at 2026-09-27 04:02:16 on dist-test-slave-glgz"
server_key: "7fceabbe5bbd7ba8cfc2e473589f7a63"
server_key_iv: "4b406bf9c54744e2e03c0fc81270c1ed"
server_key_version: "encryptionkey@0"
I20260927 04:02:16.055153 2226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/master-0/wal/instance:
uuid: "e32be32ca382450a9339506e33c983a2"
format_stamp: "Formatted at 2026-09-27 04:02:16 on dist-test-slave-glgz"
server_key: "7fceabbe5bbd7ba8cfc2e473589f7a63"
server_key_iv: "4b406bf9c54744e2e03c0fc81270c1ed"
server_key_version: "encryptionkey@0"
I20260927 04:02:16.056459 2226 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:02:16.057195 2240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:16.057379 2226 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 04:02:16.057469 2226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/master-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/master-0/wal
uuid: "e32be32ca382450a9339506e33c983a2"
format_stamp: "Formatted at 2026-09-27 04:02:16 on dist-test-slave-glgz"
server_key: "7fceabbe5bbd7ba8cfc2e473589f7a63"
server_key_iv: "4b406bf9c54744e2e03c0fc81270c1ed"
server_key_version: "encryptionkey@0"
I20260927 04:02:16.057557 2226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:02:16.064769 2226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:02:16.065095 2226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:02:16.065249 2226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:02:16.069126 2226 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.126:36163
I20260927 04:02:16.069175 2292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.126:36163 every 8 connection(s)
I20260927 04:02:16.069583 2226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
I20260927 04:02:16.070600 2293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:02:16.072949 2293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e32be32ca382450a9339506e33c983a2: Bootstrap starting.
I20260927 04:02:16.073549 2293 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e32be32ca382450a9339506e33c983a2: Neither blocks nor log segments found. Creating new log.
I20260927 04:02:16.073915 2293 log.cc:824] T 00000000000000000000000000000000 P e32be32ca382450a9339506e33c983a2: Log is configured to *not* fsync() on all Append() calls
I20260927 04:02:16.073932 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 2226
I20260927 04:02:16.074044 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/master-0/wal/instance
I20260927 04:02:16.074239 16101 external_mini_cluster.cc:1428] Setting key 55e4819471975182e5e8ce5972b55049
I20260927 04:02:16.074775 2293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e32be32ca382450a9339506e33c983a2: No bootstrap required, opened a new log
I20260927 04:02:16.077224 2293 raft_consensus.cc:348] T 00000000000000000000000000000000 P e32be32ca382450a9339506e33c983a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e32be32ca382450a9339506e33c983a2" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 36163 } }
I20260927 04:02:16.077361 2293 raft_consensus.cc:374] T 00000000000000000000000000000000 P e32be32ca382450a9339506e33c983a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:02:16.077384 2293 raft_consensus.cc:729] T 00000000000000000000000000000000 P e32be32ca382450a9339506e33c983a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e32be32ca382450a9339506e33c983a2, State: Initialized, Role: FOLLOWER
I20260927 04:02:16.077513 2293 consensus_queue.cc:260] T 00000000000000000000000000000000 P e32be32ca382450a9339506e33c983a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e32be32ca382450a9339506e33c983a2" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 36163 } }
I20260927 04:02:16.077605 2293 raft_consensus.cc:388] T 00000000000000000000000000000000 P e32be32ca382450a9339506e33c983a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:02:16.077643 2293 raft_consensus.cc:482] T 00000000000000000000000000000000 P e32be32ca382450a9339506e33c983a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:02:16.077692 2293 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e32be32ca382450a9339506e33c983a2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:02:16.078233 2293 raft_consensus.cc:504] T 00000000000000000000000000000000 P e32be32ca382450a9339506e33c983a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e32be32ca382450a9339506e33c983a2" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 36163 } }
I20260927 04:02:16.078347 2293 leader_election.cc:302] T 00000000000000000000000000000000 P e32be32ca382450a9339506e33c983a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e32be32ca382450a9339506e33c983a2; no voters:
I20260927 04:02:16.078537 2293 leader_election.cc:288] T 00000000000000000000000000000000 P e32be32ca382450a9339506e33c983a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:02:16.078567 2299 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e32be32ca382450a9339506e33c983a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:02:16.078699 2299 raft_consensus.cc:686] T 00000000000000000000000000000000 P e32be32ca382450a9339506e33c983a2 [term 1 LEADER]: Becoming Leader. State: Replica: e32be32ca382450a9339506e33c983a2, State: Running, Role: LEADER
I20260927 04:02:16.078811 2293 sys_catalog.cc:564] T 00000000000000000000000000000000 P e32be32ca382450a9339506e33c983a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:02:16.078840 2299 consensus_queue.cc:237] T 00000000000000000000000000000000 P e32be32ca382450a9339506e33c983a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e32be32ca382450a9339506e33c983a2" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 36163 } }
I20260927 04:02:16.079855 2300 sys_catalog.cc:455] T 00000000000000000000000000000000 P e32be32ca382450a9339506e33c983a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e32be32ca382450a9339506e33c983a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e32be32ca382450a9339506e33c983a2" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 36163 } } }
I20260927 04:02:16.079954 2300 sys_catalog.cc:458] T 00000000000000000000000000000000 P e32be32ca382450a9339506e33c983a2 [sys.catalog]: This master's current role is: LEADER
I20260927 04:02:16.080186 2301 sys_catalog.cc:455] T 00000000000000000000000000000000 P e32be32ca382450a9339506e33c983a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e32be32ca382450a9339506e33c983a2. Latest consensus state: current_term: 1 leader_uuid: "e32be32ca382450a9339506e33c983a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e32be32ca382450a9339506e33c983a2" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 36163 } } }
I20260927 04:02:16.080339 2301 sys_catalog.cc:458] T 00000000000000000000000000000000 P e32be32ca382450a9339506e33c983a2 [sys.catalog]: This master's current role is: LEADER
W20260927 04:02:16.080739 2315 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e32be32ca382450a9339506e33c983a2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260927 04:02:16.080755 2297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:16.074793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37592 (local address 127.15.185.126:36163)
0927 04:02:16.074977 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:16.074985 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:16.075008 (+ 23us) server_negotiation.cc:407] Connection header received
0927 04:02:16.076000 (+ 992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:16.076008 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:16.076068 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:16.076183 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:16.076434 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:16.077344 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:16.078247 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:16.078851 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:16.079352 (+ 501us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:16.079800 (+ 448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:16.079804 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:16.079804 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:02:16.080016 (+ 212us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:16.080061 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:02:16.080075 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:16.080166 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:16.080258 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 04:02:16.080527 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":74,"threads_started":1}
W20260927 04:02:16.080832 2315 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 04:02:16.080886 2312 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:02:16.081044 2312 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:02:16.082563 2312 catalog_manager.cc:1347] Generated new cluster ID: ba91937d13144a1c8e6764a4bf412482
I20260927 04:02:16.082638 2312 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:02:16.096464 2312 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:02:16.097066 2312 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:02:16.103785 2312 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e32be32ca382450a9339506e33c983a2: Generated new TSK 0
I20260927 04:02:16.104022 2312 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 04:02:16.112524 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.65:0
--local_ip_for_outbound_sockets=127.15.185.65
--webserver_interface=127.15.185.65
--webserver_port=0
--tserver_master_addrs=127.15.185.126:36163
--builtin_ntp_servers=127.15.185.84:43055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 04:02:16.197786 2318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:02:16.198009 2318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:02:16.198031 2318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:02:16.199348 2318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:02:16.199394 2318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:02:16.199421 2318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.65
I20260927 04:02:16.200831 2318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:43055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.185.65
--webserver_port=0
--tserver_master_addrs=127.15.185.126:36163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.65
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:02:16.201085 2318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:02:16.201398 2318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:02:16.207510 2324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:16.207521 2323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:16.207799 2318 server_base.cc:1034] running on GCE node
W20260927 04:02:16.207521 2326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:16.208145 2318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:02:16.208424 2318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:02:16.209585 2318 hybrid_clock.cc:648] HybridClock initialized: now 1790481736209563 us; error 39 us; skew 500 ppm
I20260927 04:02:16.210971 2318 webserver.cc:460] Webserver started at http://127.15.185.65:46167/ using document root <none> and password file <none>
I20260927 04:02:16.211267 2318 fs_manager.cc:359] Metadata directory not provided
I20260927 04:02:16.211341 2318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:02:16.211494 2318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:02:16.212589 2318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/ts-0/data/instance:
uuid: "d089d377956847c3959e19d17db8069b"
format_stamp: "Formatted at 2026-09-27 04:02:16 on dist-test-slave-glgz"
server_key: "7fe10305a2cad619ee2f7dc5036f0d4c"
server_key_iv: "2347fc25b8c058bf7fa1b2097d7ca8bf"
server_key_version: "encryptionkey@0"
I20260927 04:02:16.212954 2318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/ts-0/wal/instance:
uuid: "d089d377956847c3959e19d17db8069b"
format_stamp: "Formatted at 2026-09-27 04:02:16 on dist-test-slave-glgz"
server_key: "7fe10305a2cad619ee2f7dc5036f0d4c"
server_key_iv: "2347fc25b8c058bf7fa1b2097d7ca8bf"
server_key_version: "encryptionkey@0"
I20260927 04:02:16.214356 2318 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 04:02:16.215157 2332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:16.215333 2318 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:02:16.215432 2318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/ts-0/wal
uuid: "d089d377956847c3959e19d17db8069b"
format_stamp: "Formatted at 2026-09-27 04:02:16 on dist-test-slave-glgz"
server_key: "7fe10305a2cad619ee2f7dc5036f0d4c"
server_key_iv: "2347fc25b8c058bf7fa1b2097d7ca8bf"
server_key_version: "encryptionkey@0"
I20260927 04:02:16.215512 2318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:02:16.230628 2318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:02:16.230973 2318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:02:16.231172 2318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:02:16.231423 2318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:02:16.231770 2318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:02:16.231822 2318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:16.231864 2318 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:02:16.231891 2318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:16.237727 2318 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.65:39751
I20260927 04:02:16.237778 2445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.65:39751 every 8 connection(s)
I20260927 04:02:16.238135 2318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
I20260927 04:02:16.244069 2297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:16.238946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:48593 (local address 127.15.185.126:36163)
0927 04:02:16.238998 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:16.239000 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:16.239278 (+ 278us) server_negotiation.cc:407] Connection header received
0927 04:02:16.239797 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:16.239799 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:16.239815 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:16.239892 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:16.240674 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:16.241188 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:16.242378 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:16.242638 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:16.242706 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:16.243761 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:16.243766 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:16.243766 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:02:16.243771 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:16.243788 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:02:16.243792 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:16.243842 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:16.243970 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 04:02:16.243997 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 04:02:16.244215 2448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:16.238899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:36163 (local address 127.15.185.65:48593)
0927 04:02:16.239130 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:16.239191 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:16.239680 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:16.239951 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:16.239956 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:16.239965 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:02:16.240519 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:16.240525 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:16.241309 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:16.241310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:16.242268 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:16.242271 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:16.242761 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:16.242763 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:16.242897 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:16.243599 (+ 702us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 04:02:16.243600 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:02:16.243633 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 04:02:16.243660 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:02:16.243852 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:02:16.243856 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:02:16.243859 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 04:02:16.243916 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 04:02:16.244008 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":95,"threads_started":1}
I20260927 04:02:16.244742 2446 heartbeater.cc:347] Connected to a master server at 127.15.185.126:36163
I20260927 04:02:16.244853 2446 heartbeater.cc:464] Registering TS with master...
I20260927 04:02:16.245009 2446 heartbeater.cc:511] Master 127.15.185.126:36163 requested a full tablet report, sending...
I20260927 04:02:16.245503 2257 ts_manager.cc:194] Registered new tserver with Master: d089d377956847c3959e19d17db8069b (127.15.185.65:39751)
I20260927 04:02:16.246213 2257 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.185.65:48593
I20260927 04:02:16.248003 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 2318
I20260927 04:02:16.248136 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/ts-0/wal/instance
I20260927 04:02:16.248358 16101 external_mini_cluster.cc:1428] Setting key 55cb292f88e0fc33c40557ef29452766
I20260927 04:02:16.249385 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.66:0
--local_ip_for_outbound_sockets=127.15.185.66
--webserver_interface=127.15.185.66
--webserver_port=0
--tserver_master_addrs=127.15.185.126:36163
--builtin_ntp_servers=127.15.185.84:43055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 04:02:16.332063 2449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:02:16.332388 2449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:02:16.332427 2449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:02:16.333736 2449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:02:16.333815 2449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:02:16.333858 2449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.66
I20260927 04:02:16.335325 2449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:43055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.185.66
--webserver_port=0
--tserver_master_addrs=127.15.185.126:36163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.66
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:02:16.335698 2449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:02:16.336035 2449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:02:16.341229 2454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:16.341325 2449 server_base.cc:1034] running on GCE node
W20260927 04:02:16.341217 2455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:16.341214 2457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:16.341580 2449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:02:16.341825 2449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:02:16.342968 2449 hybrid_clock.cc:648] HybridClock initialized: now 1790481736342955 us; error 29 us; skew 500 ppm
I20260927 04:02:16.344302 2449 webserver.cc:460] Webserver started at http://127.15.185.66:42293/ using document root <none> and password file <none>
I20260927 04:02:16.344519 2449 fs_manager.cc:359] Metadata directory not provided
I20260927 04:02:16.344561 2449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:02:16.344652 2449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:02:16.345737 2449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/ts-1/data/instance:
uuid: "c4bb3c0f9b264194a8eab6b261b39602"
format_stamp: "Formatted at 2026-09-27 04:02:16 on dist-test-slave-glgz"
server_key: "5b930575aa4a6b28a7dfd060c178b49a"
server_key_iv: "97232aa63ce784856db36bf24284384f"
server_key_version: "encryptionkey@0"
I20260927 04:02:16.346035 2449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/ts-1/wal/instance:
uuid: "c4bb3c0f9b264194a8eab6b261b39602"
format_stamp: "Formatted at 2026-09-27 04:02:16 on dist-test-slave-glgz"
server_key: "5b930575aa4a6b28a7dfd060c178b49a"
server_key_iv: "97232aa63ce784856db36bf24284384f"
server_key_version: "encryptionkey@0"
I20260927 04:02:16.347470 2449 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 04:02:16.348215 2463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:16.348393 2449 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 04:02:16.348497 2449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/ts-1/wal
uuid: "c4bb3c0f9b264194a8eab6b261b39602"
format_stamp: "Formatted at 2026-09-27 04:02:16 on dist-test-slave-glgz"
server_key: "5b930575aa4a6b28a7dfd060c178b49a"
server_key_iv: "97232aa63ce784856db36bf24284384f"
server_key_version: "encryptionkey@0"
I20260927 04:02:16.348573 2449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:02:16.362560 2449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:02:16.362913 2449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:02:16.363121 2449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:02:16.363379 2449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:02:16.363727 2449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:02:16.363780 2449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:16.363824 2449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:02:16.363855 2449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:16.369695 2449 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.66:37585
I20260927 04:02:16.369736 2578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.66:37585 every 8 connection(s)
I20260927 04:02:16.370080 2449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
I20260927 04:02:16.374727 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 2449
I20260927 04:02:16.374851 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/ts-1/wal/instance
I20260927 04:02:16.375082 16101 external_mini_cluster.cc:1428] Setting key 71b92f5f806041028df5fa4aeb529eb0
I20260927 04:02:16.375790 2297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:16.370835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:43821 (local address 127.15.185.126:36163)
0927 04:02:16.370904 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:16.370906 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:16.371219 (+ 313us) server_negotiation.cc:407] Connection header received
0927 04:02:16.371761 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:16.371762 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:16.371774 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:16.371831 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:16.372582 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:16.373124 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:16.374138 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:16.374336 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:16.374407 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:16.375441 (+ 1034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:16.375456 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:16.375457 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:02:16.375463 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:16.375480 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:02:16.375485 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:16.375544 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:16.375606 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 04:02:16.375638 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 04:02:16.375963 2581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:16.370820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:36163 (local address 127.15.185.66:43821)
0927 04:02:16.371023 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:16.371090 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:16.371654 (+ 564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:16.371904 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:16.371908 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:16.371918 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:02:16.372436 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:16.372440 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:16.373226 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:16.373227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:16.374036 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:16.374039 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:16.374417 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:16.374419 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:16.374580 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:16.375262 (+ 682us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 04:02:16.375263 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:02:16.375309 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 04:02:16.375333 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:02:16.375545 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:02:16.375549 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:02:16.375552 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 04:02:16.375602 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 04:02:16.375711 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":86,"threads_started":1}
I20260927 04:02:16.376154 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.67:0
--local_ip_for_outbound_sockets=127.15.185.67
--webserver_interface=127.15.185.67
--webserver_port=0
--tserver_master_addrs=127.15.185.126:36163
--builtin_ntp_servers=127.15.185.84:43055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 04:02:16.377733 2579 heartbeater.cc:347] Connected to a master server at 127.15.185.126:36163
I20260927 04:02:16.377857 2579 heartbeater.cc:464] Registering TS with master...
I20260927 04:02:16.378048 2579 heartbeater.cc:511] Master 127.15.185.126:36163 requested a full tablet report, sending...
I20260927 04:02:16.378463 2257 ts_manager.cc:194] Registered new tserver with Master: c4bb3c0f9b264194a8eab6b261b39602 (127.15.185.66:37585)
I20260927 04:02:16.379029 2257 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.185.66:43821
W20260927 04:02:16.459414 2582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:02:16.459671 2582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:02:16.459704 2582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:02:16.460934 2582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:02:16.460991 2582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:02:16.461033 2582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.67
I20260927 04:02:16.462433 2582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:43055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.185.67
--webserver_port=0
--tserver_master_addrs=127.15.185.126:36163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.67
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:02:16.462716 2582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:02:16.463032 2582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:02:16.468000 2590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:16.467994 2587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:16.468108 2588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:16.468333 2582 server_base.cc:1034] running on GCE node
I20260927 04:02:16.468560 2582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:02:16.468789 2582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:02:16.469940 2582 hybrid_clock.cc:648] HybridClock initialized: now 1790481736469923 us; error 35 us; skew 500 ppm
I20260927 04:02:16.471335 2582 webserver.cc:460] Webserver started at http://127.15.185.67:41097/ using document root <none> and password file <none>
I20260927 04:02:16.471580 2582 fs_manager.cc:359] Metadata directory not provided
I20260927 04:02:16.471645 2582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:02:16.471784 2582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:02:16.472857 2582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/ts-2/data/instance:
uuid: "01ee6edb63314a2c86035ba68f48f18e"
format_stamp: "Formatted at 2026-09-27 04:02:16 on dist-test-slave-glgz"
server_key: "705045e82bd3704d380bd415f0dc91e4"
server_key_iv: "e3f086f1d3946bf822c0960d24e70d0c"
server_key_version: "encryptionkey@0"
I20260927 04:02:16.473207 2582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/ts-2/wal/instance:
uuid: "01ee6edb63314a2c86035ba68f48f18e"
format_stamp: "Formatted at 2026-09-27 04:02:16 on dist-test-slave-glgz"
server_key: "705045e82bd3704d380bd415f0dc91e4"
server_key_iv: "e3f086f1d3946bf822c0960d24e70d0c"
server_key_version: "encryptionkey@0"
I20260927 04:02:16.474504 2582 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 04:02:16.475246 2596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:16.475414 2582 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 04:02:16.475507 2582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/ts-2/wal
uuid: "01ee6edb63314a2c86035ba68f48f18e"
format_stamp: "Formatted at 2026-09-27 04:02:16 on dist-test-slave-glgz"
server_key: "705045e82bd3704d380bd415f0dc91e4"
server_key_iv: "e3f086f1d3946bf822c0960d24e70d0c"
server_key_version: "encryptionkey@0"
I20260927 04:02:16.475582 2582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:02:16.482358 2582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:02:16.482646 2582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:02:16.482788 2582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:02:16.482991 2582 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:02:16.483340 2582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:02:16.483392 2582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:16.483425 2582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:02:16.483453 2582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:16.489252 2582 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.67:43029
I20260927 04:02:16.489298 2710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.67:43029 every 8 connection(s)
I20260927 04:02:16.489626 2582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
I20260927 04:02:16.491422 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 2582
I20260927 04:02:16.491536 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/ts-2/wal/instance
I20260927 04:02:16.491750 16101 external_mini_cluster.cc:1428] Setting key 5a7a6fc201f95a671221fe3fdaf6bbce
I20260927 04:02:16.495258 2297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:16.490359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:53941 (local address 127.15.185.126:36163)
0927 04:02:16.490443 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:16.490446 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:16.490674 (+ 228us) server_negotiation.cc:407] Connection header received
0927 04:02:16.491262 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:16.491264 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:16.491275 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:16.491341 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:16.492249 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:16.492799 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:16.493818 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:16.494015 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:16.494088 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:16.494950 (+ 862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:16.494951 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:16.494952 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:02:16.494957 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:16.494972 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:02:16.494981 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:16.495038 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:16.495141 (+ 103us) server_negotiation.cc:299] Negotiation successful
0927 04:02:16.495170 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 04:02:16.495481 2713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:16.490345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:36163 (local address 127.15.185.67:53941)
0927 04:02:16.490546 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:16.490589 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:16.491148 (+ 559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:16.491425 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:16.491430 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:16.491441 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:02:16.492165 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:16.492173 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:16.492892 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:16.492893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:16.493723 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:16.493725 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:16.494095 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:16.494096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:16.494245 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:16.494796 (+ 551us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 04:02:16.494796 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:02:16.494826 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 04:02:16.494849 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:02:16.495039 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:02:16.495066 (+ 27us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:02:16.495069 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 04:02:16.495144 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 04:02:16.495251 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":100,"threads_started":1}
I20260927 04:02:16.495965 2711 heartbeater.cc:347] Connected to a master server at 127.15.185.126:36163
I20260927 04:02:16.496095 2711 heartbeater.cc:464] Registering TS with master...
I20260927 04:02:16.496308 2711 heartbeater.cc:511] Master 127.15.185.126:36163 requested a full tablet report, sending...
I20260927 04:02:16.496691 2257 ts_manager.cc:194] Registered new tserver with Master: 01ee6edb63314a2c86035ba68f48f18e (127.15.185.67:43029)
I20260927 04:02:16.497090 2257 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.185.67:53941
I20260927 04:02:16.503728 16101 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 04:02:16.508169 2297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:16.505079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37602 (local address 127.15.185.126:36163)
0927 04:02:16.505142 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:16.505144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:16.505270 (+ 126us) server_negotiation.cc:407] Connection header received
0927 04:02:16.505348 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:16.505349 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:16.505363 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:16.505422 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:16.505581 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:16.506089 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:16.506861 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:16.507082 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:16.507164 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:16.507877 (+ 713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:16.507878 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:16.507879 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:02:16.507883 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:16.507897 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:02:16.507901 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:16.508008 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:16.508073 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 04:02:16.508083 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 04:02:16.509850 2257 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37602:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 04:02:16.511550 2257 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 04:02:16.518159 2723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:16.513849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48130 (local address 127.15.185.65:39751)
0927 04:02:16.514029 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:16.514033 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:16.514558 (+ 525us) server_negotiation.cc:407] Connection header received
0927 04:02:16.514697 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:16.514699 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:16.514715 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:16.514785 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:02:16.514963 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:16.516030 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:16.517205 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:16.517766 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:16.517848 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:16.517936 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:16.518018 (+ 82us) server_negotiation.cc:299] Negotiation successful
0927 04:02:16.518058 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":95,"threads_started":1}
I20260927 04:02:16.518181 2724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:16.513889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49152 (local address 127.15.185.66:37585)
0927 04:02:16.514088 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:16.514092 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:16.514245 (+ 153us) server_negotiation.cc:407] Connection header received
0927 04:02:16.514328 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:16.514330 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:16.514346 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:16.514429 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:02:16.514679 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:16.515570 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:16.517037 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:16.517737 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:16.517797 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:16.517872 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:16.518047 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 04:02:16.518090 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":83,"threads_started":1}
I20260927 04:02:16.518410 2722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:16.513659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34378 (local address 127.15.185.67:43029)
0927 04:02:16.513812 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:16.513817 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:16.514108 (+ 291us) server_negotiation.cc:407] Connection header received
0927 04:02:16.514165 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:16.514168 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:16.514191 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:16.514315 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:02:16.514495 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:16.515647 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:16.516879 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:16.517620 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:16.517692 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:16.517795 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:16.518281 (+ 486us) server_negotiation.cc:299] Negotiation successful
0927 04:02:16.518329 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":80,"threads_started":1}
I20260927 04:02:16.518179 2719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:16.513594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.65:39751 (local address 127.0.0.1:48130)
0927 04:02:16.514314 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:16.514323 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:16.514590 (+ 267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:16.514846 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:16.514847 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:16.514849 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:02:16.514886 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:16.514888 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:16.516110 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:16.516111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:16.517126 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:16.517127 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:16.517855 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:16.517856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:16.517919 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:16.517941 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 04:02:16.517993 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 04:02:16.518062 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"mutex_wait_us":3,"thread_start_us":91,"threads_started":1}
I20260927 04:02:16.518558 2720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:16.513581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.67:43029 (local address 127.0.0.1:34378)
0927 04:02:16.513883 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:16.513928 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:16.514008 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:16.514327 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:16.514331 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:16.514333 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:02:16.514407 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:16.514409 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:16.515752 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:16.515753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:16.516795 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:16.516798 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:16.517831 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:16.517832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:16.517913 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:16.517926 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 04:02:16.518000 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 04:02:16.518224 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":178,"thread_start_us":71,"threads_started":1}
I20260927 04:02:16.519341 2645 tablet_service.cc:1467] Processing CreateTablet for tablet 93277c848a8a4359a2def41692320a10 (DEFAULT_TABLE table=test-table [id=6c4fabf456264c2e8c46756111462fb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:02:16.519354 2380 tablet_service.cc:1467] Processing CreateTablet for tablet 93277c848a8a4359a2def41692320a10 (DEFAULT_TABLE table=test-table [id=6c4fabf456264c2e8c46756111462fb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:02:16.519701 2380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93277c848a8a4359a2def41692320a10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:02:16.519701 2645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93277c848a8a4359a2def41692320a10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:02:16.522274 2725 tablet_bootstrap.cc:492] T 93277c848a8a4359a2def41692320a10 P d089d377956847c3959e19d17db8069b: Bootstrap starting.
I20260927 04:02:16.518275 2721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:16.513594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.66:37585 (local address 127.0.0.1:49152)
0927 04:02:16.514165 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:16.514176 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:16.514237 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:16.514451 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:16.514452 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:16.514454 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:02:16.514523 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:16.514525 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:16.515661 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:16.515662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:16.516948 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:16.516951 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:16.517902 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:16.517902 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:16.517952 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:16.517955 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 04:02:16.518000 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 04:02:16.518111 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"mutex_wait_us":114,"thread_start_us":92,"threads_started":1}
I20260927 04:02:16.522990 2725 tablet_bootstrap.cc:654] T 93277c848a8a4359a2def41692320a10 P d089d377956847c3959e19d17db8069b: Neither blocks nor log segments found. Creating new log.
I20260927 04:02:16.523206 2726 tablet_bootstrap.cc:492] T 93277c848a8a4359a2def41692320a10 P 01ee6edb63314a2c86035ba68f48f18e: Bootstrap starting.
I20260927 04:02:16.523176 2513 tablet_service.cc:1467] Processing CreateTablet for tablet 93277c848a8a4359a2def41692320a10 (DEFAULT_TABLE table=test-table [id=6c4fabf456264c2e8c46756111462fb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:02:16.523351 2725 log.cc:824] T 93277c848a8a4359a2def41692320a10 P d089d377956847c3959e19d17db8069b: Log is configured to *not* fsync() on all Append() calls
I20260927 04:02:16.523525 2513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93277c848a8a4359a2def41692320a10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:02:16.523823 2726 tablet_bootstrap.cc:654] T 93277c848a8a4359a2def41692320a10 P 01ee6edb63314a2c86035ba68f48f18e: Neither blocks nor log segments found. Creating new log.
I20260927 04:02:16.524080 2726 log.cc:824] T 93277c848a8a4359a2def41692320a10 P 01ee6edb63314a2c86035ba68f48f18e: Log is configured to *not* fsync() on all Append() calls
I20260927 04:02:16.524204 2725 tablet_bootstrap.cc:492] T 93277c848a8a4359a2def41692320a10 P d089d377956847c3959e19d17db8069b: No bootstrap required, opened a new log
I20260927 04:02:16.524264 2725 ts_tablet_manager.cc:1397] T 93277c848a8a4359a2def41692320a10 P d089d377956847c3959e19d17db8069b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 04:02:16.524670 2726 tablet_bootstrap.cc:492] T 93277c848a8a4359a2def41692320a10 P 01ee6edb63314a2c86035ba68f48f18e: No bootstrap required, opened a new log
I20260927 04:02:16.524713 2726 ts_tablet_manager.cc:1397] T 93277c848a8a4359a2def41692320a10 P 01ee6edb63314a2c86035ba68f48f18e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 04:02:16.526882 2730 tablet_bootstrap.cc:492] T 93277c848a8a4359a2def41692320a10 P c4bb3c0f9b264194a8eab6b261b39602: Bootstrap starting.
I20260927 04:02:16.526785 2726 raft_consensus.cc:348] T 93277c848a8a4359a2def41692320a10 P 01ee6edb63314a2c86035ba68f48f18e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4bb3c0f9b264194a8eab6b261b39602" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 37585 } } peers { permanent_uuid: "01ee6edb63314a2c86035ba68f48f18e" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 43029 } } peers { permanent_uuid: "d089d377956847c3959e19d17db8069b" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 39751 } }
I20260927 04:02:16.526948 2726 raft_consensus.cc:374] T 93277c848a8a4359a2def41692320a10 P 01ee6edb63314a2c86035ba68f48f18e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:02:16.526981 2726 raft_consensus.cc:729] T 93277c848a8a4359a2def41692320a10 P 01ee6edb63314a2c86035ba68f48f18e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01ee6edb63314a2c86035ba68f48f18e, State: Initialized, Role: FOLLOWER
I20260927 04:02:16.526983 2725 raft_consensus.cc:348] T 93277c848a8a4359a2def41692320a10 P d089d377956847c3959e19d17db8069b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4bb3c0f9b264194a8eab6b261b39602" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 37585 } } peers { permanent_uuid: "01ee6edb63314a2c86035ba68f48f18e" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 43029 } } peers { permanent_uuid: "d089d377956847c3959e19d17db8069b" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 39751 } }
I20260927 04:02:16.527138 2726 consensus_queue.cc:260] T 93277c848a8a4359a2def41692320a10 P 01ee6edb63314a2c86035ba68f48f18e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4bb3c0f9b264194a8eab6b261b39602" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 37585 } } peers { permanent_uuid: "01ee6edb63314a2c86035ba68f48f18e" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 43029 } } peers { permanent_uuid: "d089d377956847c3959e19d17db8069b" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 39751 } }
I20260927 04:02:16.527201 2725 raft_consensus.cc:374] T 93277c848a8a4359a2def41692320a10 P d089d377956847c3959e19d17db8069b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:02:16.527248 2725 raft_consensus.cc:729] T 93277c848a8a4359a2def41692320a10 P d089d377956847c3959e19d17db8069b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d089d377956847c3959e19d17db8069b, State: Initialized, Role: FOLLOWER
I20260927 04:02:16.527371 2725 consensus_queue.cc:260] T 93277c848a8a4359a2def41692320a10 P d089d377956847c3959e19d17db8069b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4bb3c0f9b264194a8eab6b261b39602" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 37585 } } peers { permanent_uuid: "01ee6edb63314a2c86035ba68f48f18e" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 43029 } } peers { permanent_uuid: "d089d377956847c3959e19d17db8069b" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 39751 } }
I20260927 04:02:16.527480 2726 ts_tablet_manager.cc:1428] T 93277c848a8a4359a2def41692320a10 P 01ee6edb63314a2c86035ba68f48f18e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 04:02:16.527542 2730 tablet_bootstrap.cc:654] T 93277c848a8a4359a2def41692320a10 P c4bb3c0f9b264194a8eab6b261b39602: Neither blocks nor log segments found. Creating new log.
I20260927 04:02:16.527541 2711 heartbeater.cc:503] Master 127.15.185.126:36163 was elected leader, sending a full tablet report...
I20260927 04:02:16.527676 2725 ts_tablet_manager.cc:1428] T 93277c848a8a4359a2def41692320a10 P d089d377956847c3959e19d17db8069b: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260927 04:02:16.527809 2446 heartbeater.cc:503] Master 127.15.185.126:36163 was elected leader, sending a full tablet report...
I20260927 04:02:16.527940 2730 log.cc:824] T 93277c848a8a4359a2def41692320a10 P c4bb3c0f9b264194a8eab6b261b39602: Log is configured to *not* fsync() on all Append() calls
I20260927 04:02:16.528671 2730 tablet_bootstrap.cc:492] T 93277c848a8a4359a2def41692320a10 P c4bb3c0f9b264194a8eab6b261b39602: No bootstrap required, opened a new log
I20260927 04:02:16.528769 2730 ts_tablet_manager.cc:1397] T 93277c848a8a4359a2def41692320a10 P c4bb3c0f9b264194a8eab6b261b39602: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 04:02:16.530730 2730 raft_consensus.cc:348] T 93277c848a8a4359a2def41692320a10 P c4bb3c0f9b264194a8eab6b261b39602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4bb3c0f9b264194a8eab6b261b39602" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 37585 } } peers { permanent_uuid: "01ee6edb63314a2c86035ba68f48f18e" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 43029 } } peers { permanent_uuid: "d089d377956847c3959e19d17db8069b" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 39751 } }
I20260927 04:02:16.530858 2730 raft_consensus.cc:374] T 93277c848a8a4359a2def41692320a10 P c4bb3c0f9b264194a8eab6b261b39602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:02:16.530880 2730 raft_consensus.cc:729] T 93277c848a8a4359a2def41692320a10 P c4bb3c0f9b264194a8eab6b261b39602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4bb3c0f9b264194a8eab6b261b39602, State: Initialized, Role: FOLLOWER
I20260927 04:02:16.531020 2730 consensus_queue.cc:260] T 93277c848a8a4359a2def41692320a10 P c4bb3c0f9b264194a8eab6b261b39602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4bb3c0f9b264194a8eab6b261b39602" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 37585 } } peers { permanent_uuid: "01ee6edb63314a2c86035ba68f48f18e" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 43029 } } peers { permanent_uuid: "d089d377956847c3959e19d17db8069b" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 39751 } }
I20260927 04:02:16.531323 2730 ts_tablet_manager.cc:1428] T 93277c848a8a4359a2def41692320a10 P c4bb3c0f9b264194a8eab6b261b39602: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 04:02:16.531356 2579 heartbeater.cc:503] Master 127.15.185.126:36163 was elected leader, sending a full tablet report...
W20260927 04:02:16.620613 2580 tablet.cc:2369] T 93277c848a8a4359a2def41692320a10 P c4bb3c0f9b264194a8eab6b261b39602: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 04:02:16.666855 2734 raft_consensus.cc:482] T 93277c848a8a4359a2def41692320a10 P c4bb3c0f9b264194a8eab6b261b39602 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:02:16.667029 2734 raft_consensus.cc:504] T 93277c848a8a4359a2def41692320a10 P c4bb3c0f9b264194a8eab6b261b39602 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4bb3c0f9b264194a8eab6b261b39602" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 37585 } } peers { permanent_uuid: "01ee6edb63314a2c86035ba68f48f18e" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 43029 } } peers { permanent_uuid: "d089d377956847c3959e19d17db8069b" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 39751 } }
I20260927 04:02:16.667469 2734 leader_election.cc:288] T 93277c848a8a4359a2def41692320a10 P c4bb3c0f9b264194a8eab6b261b39602 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 01ee6edb63314a2c86035ba68f48f18e (127.15.185.67:43029), d089d377956847c3959e19d17db8069b (127.15.185.65:39751)
I20260927 04:02:16.671468 2722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:16.667743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:41211 (local address 127.15.185.67:43029)
0927 04:02:16.667805 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:16.667808 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:16.667820 (+ 12us) server_negotiation.cc:407] Connection header received
0927 04:02:16.667937 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:16.667938 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:16.667952 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:16.668046 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:02:16.668238 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:16.668985 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:16.670508 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:16.671094 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:16.671177 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:16.671186 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:16.671349 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 04:02:16.671390 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 04:02:16.671468 2723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:16.667752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:38413 (local address 127.15.185.65:39751)
0927 04:02:16.667817 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:16.667819 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:16.668003 (+ 184us) server_negotiation.cc:407] Connection header received
0927 04:02:16.668048 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:16.668049 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:16.668061 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:16.668117 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:02:16.668265 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:16.669002 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:16.670508 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:16.671122 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:16.671186 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:16.671193 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:16.671349 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 04:02:16.671390 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 04:02:16.671482 2581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:16.667660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.67:43029 (local address 127.15.185.66:41211)
0927 04:02:16.667759 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:16.667772 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:16.667824 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:16.668059 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:16.668060 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:16.668062 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:02:16.668154 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:16.668157 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:16.669072 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:16.669073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:16.670406 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:16.670410 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:16.671190 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:16.671192 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:16.671269 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:16.671274 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 04:02:16.671336 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 04:02:16.671389 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79,"mutex_wait_us":49}
I20260927 04:02:16.671489 2735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:16.667660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.65:39751 (local address 127.15.185.66:38413)
0927 04:02:16.667935 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:16.667950 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:16.667993 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:16.668124 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:16.668125 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:16.668127 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:02:16.668194 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:16.668195 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:16.669073 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:16.669073 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:16.670406 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:16.670410 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:16.671201 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:16.671202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:16.671280 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:16.671283 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 04:02:16.671336 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 04:02:16.671389 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"mutex_wait_us":179,"thread_start_us":111,"threads_started":1}
I20260927 04:02:16.671904 2665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93277c848a8a4359a2def41692320a10" candidate_uuid: "c4bb3c0f9b264194a8eab6b261b39602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01ee6edb63314a2c86035ba68f48f18e" is_pre_election: true
I20260927 04:02:16.672068 2665 raft_consensus.cc:2454] T 93277c848a8a4359a2def41692320a10 P 01ee6edb63314a2c86035ba68f48f18e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4bb3c0f9b264194a8eab6b261b39602 in term 0.
I20260927 04:02:16.672147 2400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93277c848a8a4359a2def41692320a10" candidate_uuid: "c4bb3c0f9b264194a8eab6b261b39602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d089d377956847c3959e19d17db8069b" is_pre_election: true
I20260927 04:02:16.672304 2400 raft_consensus.cc:2454] T 93277c848a8a4359a2def41692320a10 P d089d377956847c3959e19d17db8069b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4bb3c0f9b264194a8eab6b261b39602 in term 0.
I20260927 04:02:16.672305 2468 leader_election.cc:302] T 93277c848a8a4359a2def41692320a10 P c4bb3c0f9b264194a8eab6b261b39602 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01ee6edb63314a2c86035ba68f48f18e, c4bb3c0f9b264194a8eab6b261b39602; no voters:
I20260927 04:02:16.672477 2734 raft_consensus.cc:2781] T 93277c848a8a4359a2def41692320a10 P c4bb3c0f9b264194a8eab6b261b39602 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 04:02:16.672559 2734 raft_consensus.cc:482] T 93277c848a8a4359a2def41692320a10 P c4bb3c0f9b264194a8eab6b261b39602 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 04:02:16.672596 2734 raft_consensus.cc:3037] T 93277c848a8a4359a2def41692320a10 P c4bb3c0f9b264194a8eab6b261b39602 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:02:16.673316 2734 raft_consensus.cc:504] T 93277c848a8a4359a2def41692320a10 P c4bb3c0f9b264194a8eab6b261b39602 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4bb3c0f9b264194a8eab6b261b39602" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 37585 } } peers { permanent_uuid: "01ee6edb63314a2c86035ba68f48f18e" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 43029 } } peers { permanent_uuid: "d089d377956847c3959e19d17db8069b" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 39751 } }
I20260927 04:02:16.673471 2734 leader_election.cc:288] T 93277c848a8a4359a2def41692320a10 P c4bb3c0f9b264194a8eab6b261b39602 [CANDIDATE]: Term 1 election: Requested vote from peers 01ee6edb63314a2c86035ba68f48f18e (127.15.185.67:43029), d089d377956847c3959e19d17db8069b (127.15.185.65:39751)
I20260927 04:02:16.673636 2400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93277c848a8a4359a2def41692320a10" candidate_uuid: "c4bb3c0f9b264194a8eab6b261b39602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d089d377956847c3959e19d17db8069b"
I20260927 04:02:16.673736 2400 raft_consensus.cc:3037] T 93277c848a8a4359a2def41692320a10 P d089d377956847c3959e19d17db8069b [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:02:16.674548 2400 raft_consensus.cc:2454] T 93277c848a8a4359a2def41692320a10 P d089d377956847c3959e19d17db8069b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4bb3c0f9b264194a8eab6b261b39602 in term 1.
I20260927 04:02:16.674654 2665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93277c848a8a4359a2def41692320a10" candidate_uuid: "c4bb3c0f9b264194a8eab6b261b39602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01ee6edb63314a2c86035ba68f48f18e"
I20260927 04:02:16.674755 2665 raft_consensus.cc:3037] T 93277c848a8a4359a2def41692320a10 P 01ee6edb63314a2c86035ba68f48f18e [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:02:16.674783 2467 leader_election.cc:302] T 93277c848a8a4359a2def41692320a10 P c4bb3c0f9b264194a8eab6b261b39602 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4bb3c0f9b264194a8eab6b261b39602, d089d377956847c3959e19d17db8069b; no voters:
I20260927 04:02:16.674889 2734 raft_consensus.cc:2781] T 93277c848a8a4359a2def41692320a10 P c4bb3c0f9b264194a8eab6b261b39602 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:02:16.675063 2734 raft_consensus.cc:686] T 93277c848a8a4359a2def41692320a10 P c4bb3c0f9b264194a8eab6b261b39602 [term 1 LEADER]: Becoming Leader. State: Replica: c4bb3c0f9b264194a8eab6b261b39602, State: Running, Role: LEADER
I20260927 04:02:16.675184 2734 consensus_queue.cc:237] T 93277c848a8a4359a2def41692320a10 P c4bb3c0f9b264194a8eab6b261b39602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4bb3c0f9b264194a8eab6b261b39602" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 37585 } } peers { permanent_uuid: "01ee6edb63314a2c86035ba68f48f18e" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 43029 } } peers { permanent_uuid: "d089d377956847c3959e19d17db8069b" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 39751 } }
I20260927 04:02:16.675714 2665 raft_consensus.cc:2454] T 93277c848a8a4359a2def41692320a10 P 01ee6edb63314a2c86035ba68f48f18e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4bb3c0f9b264194a8eab6b261b39602 in term 1.
I20260927 04:02:16.675984 2256 catalog_manager.cc:5495] T 93277c848a8a4359a2def41692320a10 P c4bb3c0f9b264194a8eab6b261b39602 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4bb3c0f9b264194a8eab6b261b39602 (127.15.185.66). New cstate: current_term: 1 leader_uuid: "c4bb3c0f9b264194a8eab6b261b39602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4bb3c0f9b264194a8eab6b261b39602" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 37585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01ee6edb63314a2c86035ba68f48f18e" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 43029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d089d377956847c3959e19d17db8069b" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 39751 } health_report { overall_health: UNKNOWN } } }
I20260927 04:02:16.699712 2724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:16.697188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49156 (local address 127.15.185.66:37585)
0927 04:02:16.697266 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:16.697268 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:16.697276 (+ 8us) server_negotiation.cc:407] Connection header received
0927 04:02:16.697320 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:16.697321 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:16.697342 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:16.697432 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:02:16.697574 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:16.698041 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:16.698859 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:16.699109 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:16.699186 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:16.699337 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:02:16.699440 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:02:16.699548 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:16.699588 (+ 40us) server_negotiation.cc:299] Negotiation successful
0927 04:02:16.699597 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 04:02:16.701589 2665 raft_consensus.cc:1261] T 93277c848a8a4359a2def41692320a10 P 01ee6edb63314a2c86035ba68f48f18e [term 1 FOLLOWER]: Refusing update from remote peer c4bb3c0f9b264194a8eab6b261b39602: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:02:16.701594 2400 raft_consensus.cc:1261] T 93277c848a8a4359a2def41692320a10 P d089d377956847c3959e19d17db8069b [term 1 FOLLOWER]: Refusing update from remote peer c4bb3c0f9b264194a8eab6b261b39602: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:02:16.701959 2734 consensus_queue.cc:1035] T 93277c848a8a4359a2def41692320a10 P c4bb3c0f9b264194a8eab6b261b39602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d089d377956847c3959e19d17db8069b" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 39751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:02:16.702132 2736 consensus_queue.cc:1035] T 93277c848a8a4359a2def41692320a10 P c4bb3c0f9b264194a8eab6b261b39602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01ee6edb63314a2c86035ba68f48f18e" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 43029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:02:16.707007 2257 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37602:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:02:16.707161 2257 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37602:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:02:16.707981 2257 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e32be32ca382450a9339506e33c983a2: Sending DeleteTablet for 3 replicas of tablet 93277c848a8a4359a2def41692320a10
I20260927 04:02:16.708305 2513 tablet_service.cc:1514] Processing DeleteTablet for tablet 93277c848a8a4359a2def41692320a10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:02:16 UTC) from {username='slave'} at 127.0.0.1:49152
I20260927 04:02:16.708340 2380 tablet_service.cc:1514] Processing DeleteTablet for tablet 93277c848a8a4359a2def41692320a10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:02:16 UTC) from {username='slave'} at 127.0.0.1:48130
I20260927 04:02:16.708534 2645 tablet_service.cc:1514] Processing DeleteTablet for tablet 93277c848a8a4359a2def41692320a10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:02:16 UTC) from {username='slave'} at 127.0.0.1:34378
I20260927 04:02:16.708702 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 2318
I20260927 04:02:16.708751 2750 tablet_replica.cc:330] T 93277c848a8a4359a2def41692320a10 P 01ee6edb63314a2c86035ba68f48f18e: stopping tablet replica
I20260927 04:02:16.708878 2750 raft_consensus.cc:2229] T 93277c848a8a4359a2def41692320a10 P 01ee6edb63314a2c86035ba68f48f18e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:02:16.708940 2748 tablet_replica.cc:330] T 93277c848a8a4359a2def41692320a10 P c4bb3c0f9b264194a8eab6b261b39602: stopping tablet replica
I20260927 04:02:16.709059 2750 raft_consensus.cc:2258] T 93277c848a8a4359a2def41692320a10 P 01ee6edb63314a2c86035ba68f48f18e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:02:16.709064 2748 raft_consensus.cc:2229] T 93277c848a8a4359a2def41692320a10 P c4bb3c0f9b264194a8eab6b261b39602 [term 1 LEADER]: Raft consensus shutting down.
I20260927 04:02:16.709450 2748 raft_consensus.cc:2258] T 93277c848a8a4359a2def41692320a10 P c4bb3c0f9b264194a8eab6b261b39602 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:02:16.709872 2748 ts_tablet_manager.cc:1905] T 93277c848a8a4359a2def41692320a10 P c4bb3c0f9b264194a8eab6b261b39602: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:02:16.711548 2748 ts_tablet_manager.cc:1918] T 93277c848a8a4359a2def41692320a10 P c4bb3c0f9b264194a8eab6b261b39602: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:02:16.711670 2748 log.cc:1196] T 93277c848a8a4359a2def41692320a10 P c4bb3c0f9b264194a8eab6b261b39602: Deleting WAL directory at /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/ts-1/wal/wals/93277c848a8a4359a2def41692320a10
I20260927 04:02:16.711949 2748 ts_tablet_manager.cc:1939] T 93277c848a8a4359a2def41692320a10 P c4bb3c0f9b264194a8eab6b261b39602: Deleting consensus metadata
I20260927 04:02:16.712399 2241 catalog_manager.cc:4841] TS c4bb3c0f9b264194a8eab6b261b39602 (127.15.185.66:37585): tablet 93277c848a8a4359a2def41692320a10 (table test-table [id=6c4fabf456264c2e8c46756111462fb6]) successfully deleted
W20260927 04:02:16.714908 2243 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.185.65:39751 (error 108)
I20260927 04:02:16.716553 2721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:16.716416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.65:39751 (local address 127.0.0.1:48142)
0927 04:02:16.716475 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:16.716518 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.65:39751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260927 04:02:16.716658 2243 catalog_manager.cc:4584] TS d089d377956847c3959e19d17db8069b (127.15.185.65:39751): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 93277c848a8a4359a2def41692320a10: Network error: Client connection negotiation failed: client connection to 127.15.185.65:39751: connect: Connection refused (error 111)
I20260927 04:02:16.716765 2750 ts_tablet_manager.cc:1905] T 93277c848a8a4359a2def41692320a10 P 01ee6edb63314a2c86035ba68f48f18e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:02:16.718401 2750 ts_tablet_manager.cc:1918] T 93277c848a8a4359a2def41692320a10 P 01ee6edb63314a2c86035ba68f48f18e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:02:16.718521 2750 log.cc:1196] T 93277c848a8a4359a2def41692320a10 P 01ee6edb63314a2c86035ba68f48f18e: Deleting WAL directory at /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481498018407-16101-0/minicluster-data/ts-2/wal/wals/93277c848a8a4359a2def41692320a10
I20260927 04:02:16.718782 2750 ts_tablet_manager.cc:1939] T 93277c848a8a4359a2def41692320a10 P 01ee6edb63314a2c86035ba68f48f18e: Deleting consensus metadata
I20260927 04:02:16.719322 2244 catalog_manager.cc:4841] TS 01ee6edb63314a2c86035ba68f48f18e (127.15.185.67:43029): tablet 93277c848a8a4359a2def41692320a10 (table test-table [id=6c4fabf456264c2e8c46756111462fb6]) successfully deleted
I20260927 04:02:16.719534 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 2449
I20260927 04:02:16.724726 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 2582
I20260927 04:02:16.729305 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 2226
2026-09-27T04:02:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (796 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:02:16 dist-test-slave-glgz krb5kdc[2754](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:02:16 dist-test-slave-glgz krb5kdc[2754](info): set up 2 sockets
Sep 27 04:02:16 dist-test-slave-glgz krb5kdc[2754](info): commencing operation
krb5kdc: starting...
W20260927 04:02:17.791358 16101 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Sep 27 04:02:17 dist-test-slave-glgz krb5kdc[2754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:02:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:02:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.185.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab.
Entry for principal kudu/127.15.185.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab.
I20260927 04:02:17.915671 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:33703
--webserver_interface=127.15.185.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:37933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.185.126:33703
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:02:18.000280 2767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:02:18.000489 2767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:02:18.000511 2767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:02:18.001593 2767 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:02:18.001629 2767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:02:18.001643 2767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:02:18.001657 2767 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:02:18.001668 2767 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:02:18.003099 2767 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:37933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.185.126:33703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:33703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:02:18.003293 2767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:02:18.003469 2767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:02:18.008301 2772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:18.008298 2775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:18.008335 2773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:18.008648 2767 server_base.cc:1034] running on GCE node
I20260927 04:02:18.008895 2767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:02:18.009204 2767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:02:18.010349 2767 hybrid_clock.cc:648] HybridClock initialized: now 1790481738010336 us; error 27 us; skew 500 ppm
Sep 27 04:02:18 dist-test-slave-glgz krb5kdc[2754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481738, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:02:18.012950 2767 init.cc:377] Logged in from keytab as kudu/127.15.185.126@KRBTEST.COM (short username kudu)
I20260927 04:02:18.013451 2767 webserver.cc:460] Webserver started at http://127.15.185.126:34961/ using document root <none> and password file <none>
I20260927 04:02:18.013697 2767 fs_manager.cc:359] Metadata directory not provided
I20260927 04:02:18.013763 2767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:02:18.013906 2767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:02:18.014968 2767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/master-0/data/instance:
uuid: "a12db7fe70944faaad5ac93894d1b3ea"
format_stamp: "Formatted at 2026-09-27 04:02:18 on dist-test-slave-glgz"
server_key: "d50d9fbfe24de6f18d313ddf061b25f5"
server_key_iv: "978c2e0f3438753e032bdbc6075ec91f"
server_key_version: "encryptionkey@0"
I20260927 04:02:18.015390 2767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/master-0/wal/instance:
uuid: "a12db7fe70944faaad5ac93894d1b3ea"
format_stamp: "Formatted at 2026-09-27 04:02:18 on dist-test-slave-glgz"
server_key: "d50d9fbfe24de6f18d313ddf061b25f5"
server_key_iv: "978c2e0f3438753e032bdbc6075ec91f"
server_key_version: "encryptionkey@0"
I20260927 04:02:18.016706 2767 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:02:18.017437 2782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:18.017616 2767 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 04:02:18.017714 2767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/master-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/master-0/wal
uuid: "a12db7fe70944faaad5ac93894d1b3ea"
format_stamp: "Formatted at 2026-09-27 04:02:18 on dist-test-slave-glgz"
server_key: "d50d9fbfe24de6f18d313ddf061b25f5"
server_key_iv: "978c2e0f3438753e032bdbc6075ec91f"
server_key_version: "encryptionkey@0"
I20260927 04:02:18.017793 2767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:02:18.025111 2767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:02:18.027359 2767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:02:18.027520 2767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:02:18.031416 2767 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.126:33703
I20260927 04:02:18.031445 2834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.126:33703 every 8 connection(s)
I20260927 04:02:18.031824 2767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
I20260927 04:02:18.032738 2835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:02:18.034936 2835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a12db7fe70944faaad5ac93894d1b3ea: Bootstrap starting.
I20260927 04:02:18.035537 2835 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a12db7fe70944faaad5ac93894d1b3ea: Neither blocks nor log segments found. Creating new log.
I20260927 04:02:18.035768 2835 log.cc:824] T 00000000000000000000000000000000 P a12db7fe70944faaad5ac93894d1b3ea: Log is configured to *not* fsync() on all Append() calls
I20260927 04:02:18.036405 2835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a12db7fe70944faaad5ac93894d1b3ea: No bootstrap required, opened a new log
I20260927 04:02:18.037914 2835 raft_consensus.cc:348] T 00000000000000000000000000000000 P a12db7fe70944faaad5ac93894d1b3ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a12db7fe70944faaad5ac93894d1b3ea" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 33703 } }
I20260927 04:02:18.038066 2835 raft_consensus.cc:374] T 00000000000000000000000000000000 P a12db7fe70944faaad5ac93894d1b3ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:02:18.038091 2835 raft_consensus.cc:729] T 00000000000000000000000000000000 P a12db7fe70944faaad5ac93894d1b3ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a12db7fe70944faaad5ac93894d1b3ea, State: Initialized, Role: FOLLOWER
I20260927 04:02:18.038196 2835 consensus_queue.cc:260] T 00000000000000000000000000000000 P a12db7fe70944faaad5ac93894d1b3ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a12db7fe70944faaad5ac93894d1b3ea" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 33703 } }
I20260927 04:02:18.038249 2835 raft_consensus.cc:388] T 00000000000000000000000000000000 P a12db7fe70944faaad5ac93894d1b3ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:02:18.038269 2835 raft_consensus.cc:482] T 00000000000000000000000000000000 P a12db7fe70944faaad5ac93894d1b3ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:02:18.038338 2835 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a12db7fe70944faaad5ac93894d1b3ea [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:02:18.038857 2835 raft_consensus.cc:504] T 00000000000000000000000000000000 P a12db7fe70944faaad5ac93894d1b3ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a12db7fe70944faaad5ac93894d1b3ea" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 33703 } }
I20260927 04:02:18.038942 2835 leader_election.cc:302] T 00000000000000000000000000000000 P a12db7fe70944faaad5ac93894d1b3ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a12db7fe70944faaad5ac93894d1b3ea; no voters:
I20260927 04:02:18.039141 2835 leader_election.cc:288] T 00000000000000000000000000000000 P a12db7fe70944faaad5ac93894d1b3ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:02:18.039215 2838 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a12db7fe70944faaad5ac93894d1b3ea [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:02:18.039402 2838 raft_consensus.cc:686] T 00000000000000000000000000000000 P a12db7fe70944faaad5ac93894d1b3ea [term 1 LEADER]: Becoming Leader. State: Replica: a12db7fe70944faaad5ac93894d1b3ea, State: Running, Role: LEADER
I20260927 04:02:18.039422 2835 sys_catalog.cc:564] T 00000000000000000000000000000000 P a12db7fe70944faaad5ac93894d1b3ea [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:02:18.039489 2838 consensus_queue.cc:237] T 00000000000000000000000000000000 P a12db7fe70944faaad5ac93894d1b3ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a12db7fe70944faaad5ac93894d1b3ea" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 33703 } }
I20260927 04:02:18.039968 2840 sys_catalog.cc:455] T 00000000000000000000000000000000 P a12db7fe70944faaad5ac93894d1b3ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader a12db7fe70944faaad5ac93894d1b3ea. Latest consensus state: current_term: 1 leader_uuid: "a12db7fe70944faaad5ac93894d1b3ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a12db7fe70944faaad5ac93894d1b3ea" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 33703 } } }
I20260927 04:02:18.040053 2840 sys_catalog.cc:458] T 00000000000000000000000000000000 P a12db7fe70944faaad5ac93894d1b3ea [sys.catalog]: This master's current role is: LEADER
I20260927 04:02:18.040122 2839 sys_catalog.cc:455] T 00000000000000000000000000000000 P a12db7fe70944faaad5ac93894d1b3ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a12db7fe70944faaad5ac93894d1b3ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a12db7fe70944faaad5ac93894d1b3ea" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 33703 } } }
I20260927 04:02:18.040184 2839 sys_catalog.cc:458] T 00000000000000000000000000000000 P a12db7fe70944faaad5ac93894d1b3ea [sys.catalog]: This master's current role is: LEADER
I20260927 04:02:18.040563 2848 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:02:18.040812 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 2767
I20260927 04:02:18.040930 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/master-0/wal/instance
I20260927 04:02:18.041064 2848 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:02:18.041162 16101 external_mini_cluster.cc:1428] Setting key ff27b595c867ccdba71b17f52c310fdf
I20260927 04:02:18.042797 2848 catalog_manager.cc:1347] Generated new cluster ID: 8d5f92509b2945799a198d54801353ac
I20260927 04:02:18.042845 2848 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 04:02:18 dist-test-slave-glgz krb5kdc[2754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 04:02:18.052331 2848 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:02:18.053076 2848 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:02:18.053334 2858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:18.041714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37960 (local address 127.15.185.126:33703)
0927 04:02:18.041875 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:18.041885 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:18.042284 (+ 399us) server_negotiation.cc:407] Connection header received
0927 04:02:18.042445 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:18.042451 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:18.042494 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:18.042630 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:18.043387 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:18.044860 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:18.045901 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:18.046562 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:18.046679 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:18.049480 (+ 2801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:18.049484 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:18.049484 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:18.049497 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:18.051538 (+ 2041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:18.051959 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:18.051963 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:18.051963 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:18.052098 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:18.052326 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:18.052327 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:18.052327 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:18.052614 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:18.052790 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:18.052913 (+ 123us) server_negotiation.cc:299] Negotiation successful
0927 04:02:18.053068 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":116,"thread_start_us":71,"threads_started":1}
I20260927 04:02:18.056279 2848 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a12db7fe70944faaad5ac93894d1b3ea: Generated new TSK 0
I20260927 04:02:18.056466 2848 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.185.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab.
Entry for principal kudu/127.15.185.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab.
I20260927 04:02:18.090400 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.65:0
--local_ip_for_outbound_sockets=127.15.185.65
--webserver_interface=127.15.185.65
--webserver_port=0
--tserver_master_addrs=127.15.185.126:33703
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab
--principal=kudu/127.15.185.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:37933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:02:18.176410 2862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:02:18.176625 2862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:02:18.176649 2862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:02:18.177783 2862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:02:18.177826 2862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:02:18.177852 2862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.65
I20260927 04:02:18.179643 2862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:37933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.65.keytab
--principal=kudu/127.15.185.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.185.65
--webserver_port=0
--tserver_master_addrs=127.15.185.126:33703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.65
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:02:18.179926 2862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:02:18.180166 2862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:02:18.185390 2867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:18.185397 2871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:18.185397 2869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:18.185424 2862 server_base.cc:1034] running on GCE node
I20260927 04:02:18.185819 2862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:02:18.186017 2862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:02:18.187117 2862 hybrid_clock.cc:648] HybridClock initialized: now 1790481738187105 us; error 27 us; skew 500 ppm
Sep 27 04:02:18 dist-test-slave-glgz krb5kdc[2754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481738, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:02:18.189507 2862 init.cc:377] Logged in from keytab as kudu/127.15.185.65@KRBTEST.COM (short username kudu)
I20260927 04:02:18.189926 2862 webserver.cc:460] Webserver started at http://127.15.185.65:36695/ using document root <none> and password file <none>
I20260927 04:02:18.190150 2862 fs_manager.cc:359] Metadata directory not provided
I20260927 04:02:18.190222 2862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:02:18.190368 2862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:02:18.191498 2862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/ts-0/data/instance:
uuid: "4680d95bd48b4a57a6794c6a42d9ca68"
format_stamp: "Formatted at 2026-09-27 04:02:18 on dist-test-slave-glgz"
server_key: "692ea4b3ee2593eec7294ef6cf94cd35"
server_key_iv: "d73bc57e9bf13be37bde38e15dc62063"
server_key_version: "encryptionkey@0"
I20260927 04:02:18.191854 2862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/ts-0/wal/instance:
uuid: "4680d95bd48b4a57a6794c6a42d9ca68"
format_stamp: "Formatted at 2026-09-27 04:02:18 on dist-test-slave-glgz"
server_key: "692ea4b3ee2593eec7294ef6cf94cd35"
server_key_iv: "d73bc57e9bf13be37bde38e15dc62063"
server_key_version: "encryptionkey@0"
I20260927 04:02:18.193151 2862 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 04:02:18.193877 2878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:18.194057 2862 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 04:02:18.194160 2862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/ts-0/wal
uuid: "4680d95bd48b4a57a6794c6a42d9ca68"
format_stamp: "Formatted at 2026-09-27 04:02:18 on dist-test-slave-glgz"
server_key: "692ea4b3ee2593eec7294ef6cf94cd35"
server_key_iv: "d73bc57e9bf13be37bde38e15dc62063"
server_key_version: "encryptionkey@0"
I20260927 04:02:18.194244 2862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:02:18.203841 2862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:02:18.205690 2862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:02:18.205847 2862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:02:18.206066 2862 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:02:18.206382 2862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:02:18.206434 2862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:18.206475 2862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:02:18.206504 2862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:18.212239 2862 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.65:40297
I20260927 04:02:18.212275 2991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.65:40297 every 8 connection(s)
I20260927 04:02:18.212620 2862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
I20260927 04:02:18.215390 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 2862
I20260927 04:02:18.215514 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/ts-0/wal/instance
I20260927 04:02:18.215751 16101 external_mini_cluster.cc:1428] Setting key 43048e99c40fb9c4ed0364dce5bee71f
Sep 27 04:02:18 dist-test-slave-glgz krb5kdc[2754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481738, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.65@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 04:02:18.222661 2858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:18.213329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:47373 (local address 127.15.185.126:33703)
0927 04:02:18.213394 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:18.213395 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:18.213609 (+ 214us) server_negotiation.cc:407] Connection header received
0927 04:02:18.214131 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:18.214132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:18.214149 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:18.214216 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:18.215226 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:18.215838 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:18.216970 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:18.217169 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:18.217241 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:18.219685 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:18.219686 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:18.219687 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:18.219694 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:18.221086 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:18.221517 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:18.221518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:18.221519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:18.221602 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:18.221945 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:18.221947 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:18.221947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:18.222105 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:18.222177 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:18.222442 (+ 265us) server_negotiation.cc:299] Negotiation successful
0927 04:02:18.222552 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 04:02:18.222692 2994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:18.213311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:33703 (local address 127.15.185.65:47373)
0927 04:02:18.213477 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:18.213519 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:18.214031 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:18.214294 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:18.214298 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:18.214897 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:02:18.215118 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:18.215123 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:18.215942 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:18.215943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:18.216873 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:18.216875 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:18.217262 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:18.217263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:18.217402 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:18.217959 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:02:18.217970 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:02:18.219578 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:02:18.221186 (+ 1608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:18.221193 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:18.221195 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:18.221432 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:18.221707 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:18.221708 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:18.221708 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:18.221852 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:18.222180 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:02:18.222186 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:02:18.222313 (+ 127us) client_negotiation.cc:769] Sending connection context
0927 04:02:18.222397 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 04:02:18.222522 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":82,"threads_started":1}
I20260927 04:02:18.223371 2992 heartbeater.cc:347] Connected to a master server at 127.15.185.126:33703
I20260927 04:02:18.223481 2992 heartbeater.cc:464] Registering TS with master...
I20260927 04:02:18.223666 2992 heartbeater.cc:511] Master 127.15.185.126:33703 requested a full tablet report, sending...
I20260927 04:02:18.224212 2799 ts_manager.cc:194] Registered new tserver with Master: 4680d95bd48b4a57a6794c6a42d9ca68 (127.15.185.65:40297)
WARNING: no policy specified for kudu/127.15.185.66@KRBTEST.COM; defaulting to no policy
I20260927 04:02:18.224884 2799 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.185.65@KRBTEST.COM'} at 127.15.185.65:47373
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab.
Entry for principal kudu/127.15.185.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab.
I20260927 04:02:18.242115 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.66:0
--local_ip_for_outbound_sockets=127.15.185.66
--webserver_interface=127.15.185.66
--webserver_port=0
--tserver_master_addrs=127.15.185.126:33703
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab
--principal=kudu/127.15.185.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:37933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:02:18.323424 2997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:02:18.323644 2997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:02:18.323668 2997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:02:18.324782 2997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:02:18.324820 2997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:02:18.324846 2997 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.66
I20260927 04:02:18.326256 2997 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:37933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.66.keytab
--principal=kudu/127.15.185.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.185.66
--webserver_port=0
--tserver_master_addrs=127.15.185.126:33703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.66
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:02:18.326495 2997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:02:18.326722 2997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:02:18.331843 3002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:18.331839 3005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:18.331858 3003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:18.332125 2997 server_base.cc:1034] running on GCE node
I20260927 04:02:18.332394 2997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:02:18.332652 2997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:02:18.333813 2997 hybrid_clock.cc:648] HybridClock initialized: now 1790481738333802 us; error 29 us; skew 500 ppm
Sep 27 04:02:18 dist-test-slave-glgz krb5kdc[2754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481738, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:02:18.336334 2997 init.cc:377] Logged in from keytab as kudu/127.15.185.66@KRBTEST.COM (short username kudu)
I20260927 04:02:18.336771 2997 webserver.cc:460] Webserver started at http://127.15.185.66:33397/ using document root <none> and password file <none>
I20260927 04:02:18.337009 2997 fs_manager.cc:359] Metadata directory not provided
I20260927 04:02:18.337073 2997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:02:18.337226 2997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:02:18.338310 2997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/ts-1/data/instance:
uuid: "1c979348306b4c0cbcf7785428e860b9"
format_stamp: "Formatted at 2026-09-27 04:02:18 on dist-test-slave-glgz"
server_key: "a26ab6552a4a8dba5b3be610bd430ffc"
server_key_iv: "e45d0c34ffa1d3d02afdf4fa6e51329f"
server_key_version: "encryptionkey@0"
I20260927 04:02:18.338670 2997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/ts-1/wal/instance:
uuid: "1c979348306b4c0cbcf7785428e860b9"
format_stamp: "Formatted at 2026-09-27 04:02:18 on dist-test-slave-glgz"
server_key: "a26ab6552a4a8dba5b3be610bd430ffc"
server_key_iv: "e45d0c34ffa1d3d02afdf4fa6e51329f"
server_key_version: "encryptionkey@0"
I20260927 04:02:18.340063 2997 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260927 04:02:18.340837 3012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:18.341029 2997 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:18.341125 2997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/ts-1/wal
uuid: "1c979348306b4c0cbcf7785428e860b9"
format_stamp: "Formatted at 2026-09-27 04:02:18 on dist-test-slave-glgz"
server_key: "a26ab6552a4a8dba5b3be610bd430ffc"
server_key_iv: "e45d0c34ffa1d3d02afdf4fa6e51329f"
server_key_version: "encryptionkey@0"
I20260927 04:02:18.341218 2997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:02:18.350148 2997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:02:18.352244 2997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:02:18.352360 2997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:02:18.352533 2997 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:02:18.352782 2997 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:02:18.352813 2997 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:18.352834 2997 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:02:18.352847 2997 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:18.358516 2997 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.66:36963
I20260927 04:02:18.358601 3125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.66:36963 every 8 connection(s)
I20260927 04:02:18.358911 2997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
Sep 27 04:02:18 dist-test-slave-glgz krb5kdc[2754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481738, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.66@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 04:02:18.367233 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 2997
I20260927 04:02:18.367362 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/ts-1/wal/instance
I20260927 04:02:18.367614 16101 external_mini_cluster.cc:1428] Setting key 88409c7f0060a7907111cc3a976925d6
I20260927 04:02:18.368538 2858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:18.359632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:56523 (local address 127.15.185.126:33703)
0927 04:02:18.359704 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:18.359713 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:18.359931 (+ 218us) server_negotiation.cc:407] Connection header received
0927 04:02:18.360428 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:18.360429 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:18.360440 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:18.360504 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:18.361435 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:18.361938 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:18.362872 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:18.363157 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:18.363213 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:18.365426 (+ 2213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:18.365428 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:18.365428 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:18.365434 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:18.366952 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:18.367492 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:18.367493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:18.367493 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:18.367555 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:18.367862 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:18.367863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:18.367864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:18.368018 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:18.368096 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:18.368366 (+ 270us) server_negotiation.cc:299] Negotiation successful
0927 04:02:18.368448 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 04:02:18.368744 3128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:18.359608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:33703 (local address 127.15.185.66:56523)
0927 04:02:18.359806 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:18.359849 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:18.360344 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:18.360573 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:18.360576 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:18.361141 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:02:18.361348 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:18.361353 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:18.362016 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:18.362017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:18.362806 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:18.362808 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:18.363238 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:18.363238 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:18.363339 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:18.363849 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:02:18.363858 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:02:18.365204 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:02:18.367079 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:18.367087 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:18.367089 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:18.367409 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:18.367619 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:18.367620 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:18.367621 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:18.367763 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:18.368087 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:02:18.368092 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:02:18.368219 (+ 127us) client_negotiation.cc:769] Sending connection context
0927 04:02:18.368331 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 04:02:18.368492 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":123,"threads_started":1}
I20260927 04:02:18.369176 3126 heartbeater.cc:347] Connected to a master server at 127.15.185.126:33703
I20260927 04:02:18.369285 3126 heartbeater.cc:464] Registering TS with master...
I20260927 04:02:18.369469 3126 heartbeater.cc:511] Master 127.15.185.126:33703 requested a full tablet report, sending...
I20260927 04:02:18.369853 2799 ts_manager.cc:194] Registered new tserver with Master: 1c979348306b4c0cbcf7785428e860b9 (127.15.185.66:36963)
I20260927 04:02:18.370281 2799 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.185.66@KRBTEST.COM'} at 127.15.185.66:56523
WARNING: no policy specified for kudu/127.15.185.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab.
Entry for principal kudu/127.15.185.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab.
I20260927 04:02:18.393591 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.67:0
--local_ip_for_outbound_sockets=127.15.185.67
--webserver_interface=127.15.185.67
--webserver_port=0
--tserver_master_addrs=127.15.185.126:33703
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab
--principal=kudu/127.15.185.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:37933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:02:18.475989 3131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:02:18.476207 3131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:02:18.476229 3131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:02:18.477468 3131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:02:18.477515 3131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:02:18.477541 3131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.67
I20260927 04:02:18.478973 3131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:37933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.67.keytab
--principal=kudu/127.15.185.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.185.67
--webserver_port=0
--tserver_master_addrs=127.15.185.126:33703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3131
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.67
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:02:18.479368 3131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:02:18.479630 3131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:02:18.484776 3139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:18.484665 3137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:18.484807 3131 server_base.cc:1034] running on GCE node
W20260927 04:02:18.484665 3136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:18.485232 3131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:02:18.485479 3131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:02:18.486658 3131 hybrid_clock.cc:648] HybridClock initialized: now 1790481738486650 us; error 34 us; skew 500 ppm
Sep 27 04:02:18 dist-test-slave-glgz krb5kdc[2754](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481738, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:02:18.489107 3131 init.cc:377] Logged in from keytab as kudu/127.15.185.67@KRBTEST.COM (short username kudu)
I20260927 04:02:18.489632 3131 webserver.cc:460] Webserver started at http://127.15.185.67:42827/ using document root <none> and password file <none>
I20260927 04:02:18.489863 3131 fs_manager.cc:359] Metadata directory not provided
I20260927 04:02:18.489928 3131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:02:18.490072 3131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:02:18.491204 3131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/ts-2/data/instance:
uuid: "7a88cd1f442d4fe4bdae9d7fb8c45823"
format_stamp: "Formatted at 2026-09-27 04:02:18 on dist-test-slave-glgz"
server_key: "e93468303625ebbefef35f7b578cc55d"
server_key_iv: "95b2350f4b6bd99da8053d9485a283c3"
server_key_version: "encryptionkey@0"
I20260927 04:02:18.491582 3131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/ts-2/wal/instance:
uuid: "7a88cd1f442d4fe4bdae9d7fb8c45823"
format_stamp: "Formatted at 2026-09-27 04:02:18 on dist-test-slave-glgz"
server_key: "e93468303625ebbefef35f7b578cc55d"
server_key_iv: "95b2350f4b6bd99da8053d9485a283c3"
server_key_version: "encryptionkey@0"
I20260927 04:02:18.492888 3131 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:02:18.493592 3146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:18.493731 3131 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 04:02:18.493858 3131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/ts-2/wal
uuid: "7a88cd1f442d4fe4bdae9d7fb8c45823"
format_stamp: "Formatted at 2026-09-27 04:02:18 on dist-test-slave-glgz"
server_key: "e93468303625ebbefef35f7b578cc55d"
server_key_iv: "95b2350f4b6bd99da8053d9485a283c3"
server_key_version: "encryptionkey@0"
I20260927 04:02:18.493932 3131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:02:18.514469 3131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:02:18.516495 3131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:02:18.516674 3131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:02:18.516937 3131 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:02:18.517266 3131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:02:18.517318 3131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:18.517357 3131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:02:18.517385 3131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:18.523327 3131 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.67:43347
I20260927 04:02:18.523394 3259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.67:43347 every 8 connection(s)
I20260927 04:02:18.523737 3131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
I20260927 04:02:18.528652 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 3131
I20260927 04:02:18.528812 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/ts-2/wal/instance
I20260927 04:02:18.529018 16101 external_mini_cluster.cc:1428] Setting key c31e421a1c0fc194d4d975517da6ef77
Sep 27 04:02:18 dist-test-slave-glgz krb5kdc[2754](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481738, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.67@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 04:02:18.533262 2858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:18.524435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:49415 (local address 127.15.185.126:33703)
0927 04:02:18.524515 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:18.524518 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:18.524683 (+ 165us) server_negotiation.cc:407] Connection header received
0927 04:02:18.525198 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:18.525199 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:18.525214 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:18.525279 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:18.526273 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:18.526791 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:18.527820 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:18.528010 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:18.528082 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:18.530641 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:18.530643 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:18.530643 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:18.530649 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:18.531920 (+ 1271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:18.532286 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:18.532287 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:18.532287 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:18.532346 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:18.532559 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:18.532559 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:18.532560 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:18.532717 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:18.532788 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:18.533077 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 04:02:18.533161 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 04:02:18.533489 3262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:18.524403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:33703 (local address 127.15.185.67:49415)
0927 04:02:18.524568 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:18.524604 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:18.525078 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:18.525353 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:18.525357 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:18.525963 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:02:18.526162 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:18.526166 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:18.526872 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:18.526873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:18.527743 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:18.527745 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:18.528104 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:18.528105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:18.528249 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:18.528824 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:02:18.528834 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:02:18.530532 (+ 1698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:02:18.531989 (+ 1457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:18.531994 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:18.531995 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:18.532199 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:18.532410 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:18.532411 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:18.532411 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:18.532503 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:18.532796 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:02:18.532804 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:02:18.532942 (+ 138us) client_negotiation.cc:769] Sending connection context
0927 04:02:18.533054 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 04:02:18.533243 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":92,"threads_started":1}
I20260927 04:02:18.533974 3260 heartbeater.cc:347] Connected to a master server at 127.15.185.126:33703
I20260927 04:02:18.534057 3260 heartbeater.cc:464] Registering TS with master...
I20260927 04:02:18.534202 3260 heartbeater.cc:511] Master 127.15.185.126:33703 requested a full tablet report, sending...
I20260927 04:02:18.534507 2799 ts_manager.cc:194] Registered new tserver with Master: 7a88cd1f442d4fe4bdae9d7fb8c45823 (127.15.185.67:43347)
I20260927 04:02:18.534883 2799 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.185.67@KRBTEST.COM'} at 127.15.185.67:49415
I20260927 04:02:18.540948 16101 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 04:02:18.548105 2858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:18.542304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37970 (local address 127.15.185.126:33703)
0927 04:02:18.542344 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:18.542346 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:18.542430 (+ 84us) server_negotiation.cc:407] Connection header received
0927 04:02:18.542484 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:18.542485 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:18.542493 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:18.542542 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:18.543208 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:18.543662 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:18.544446 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:18.544622 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:18.544673 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:18.545891 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:18.545892 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:18.545892 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:18.545898 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:18.547111 (+ 1213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:18.547423 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:18.547423 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:18.547424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:18.547474 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:18.547687 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:18.547688 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:18.547688 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:18.547819 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:18.547891 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:18.547987 (+ 96us) server_negotiation.cc:299] Negotiation successful
0927 04:02:18.548042 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 04:02:18.549678 2799 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37970:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 04:02:18.551317 2799 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 04:02:18.557852 3271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:18.553709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34286 (local address 127.15.185.67:43347)
0927 04:02:18.553897 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:18.553902 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:18.554077 (+ 175us) server_negotiation.cc:407] Connection header received
0927 04:02:18.554127 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:18.554129 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:18.554321 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:18.554394 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:02:18.554596 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:18.555666 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:18.556961 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:18.557447 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:18.557534 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:18.557595 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:18.557722 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 04:02:18.557762 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":94,"threads_started":1}
I20260927 04:02:18.557842 3269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:18.553558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.67:43347 (local address 127.0.0.1:34286)
0927 04:02:18.553946 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:18.553956 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:18.554018 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:18.554462 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:18.554463 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:18.554465 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:02:18.554514 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:18.554515 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:18.555789 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:18.555790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:18.556873 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:18.556874 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:18.557534 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:18.557535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:18.557620 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:18.557634 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 04:02:18.557702 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 04:02:18.557762 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":54,"threads_started":1}
I20260927 04:02:18.558030 3273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:18.553950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50112 (local address 127.15.185.65:40297)
0927 04:02:18.554135 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:18.554139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:18.554148 (+ 9us) server_negotiation.cc:407] Connection header received
0927 04:02:18.554183 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:18.554185 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:18.554218 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:18.554294 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:02:18.554504 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:18.555289 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:18.556773 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:18.557517 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:18.557588 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:18.557642 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:18.557897 (+ 255us) server_negotiation.cc:299] Negotiation successful
0927 04:02:18.557949 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":78,"threads_started":1}
I20260927 04:02:18.558048 3268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:18.553542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.66:36963 (local address 127.0.0.1:49090)
0927 04:02:18.553836 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:18.553881 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:18.553958 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:18.554193 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:18.554195 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:18.554197 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:02:18.554273 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:18.554274 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:18.555667 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:18.555668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:18.556846 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:18.556848 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:18.557806 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:18.557807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:18.557874 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:18.557878 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 04:02:18.557938 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 04:02:18.557974 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":3,"thread_start_us":61,"threads_started":1}
I20260927 04:02:18.558157 3272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:18.553773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.65:40297 (local address 127.0.0.1:50112)
0927 04:02:18.553946 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:18.553955 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:18.553996 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:18.554377 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:18.554378 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:18.554381 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:02:18.554424 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:18.554426 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:18.555381 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:18.555382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:18.556641 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:18.556644 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:18.557679 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:18.557680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:18.557754 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:18.557758 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 04:02:18.557807 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 04:02:18.557851 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":1,"thread_start_us":41,"threads_started":1}
I20260927 04:02:18.558327 3270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:18.553632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49090 (local address 127.15.185.66:36963)
0927 04:02:18.553830 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:18.553835 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:18.554044 (+ 209us) server_negotiation.cc:407] Connection header received
0927 04:02:18.554094 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:18.554096 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:18.554108 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:18.554190 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:02:18.554352 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:18.555552 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:18.556941 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:18.557667 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:18.557727 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:18.557776 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:18.558159 (+ 383us) server_negotiation.cc:299] Negotiation successful
0927 04:02:18.558228 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":98,"threads_started":1}
I20260927 04:02:18.558995 3060 tablet_service.cc:1467] Processing CreateTablet for tablet 13a6de508dd443eb8ac00ba0c8383761 (DEFAULT_TABLE table=test-table [id=2177f52914114ef39b4d0de4506eb825]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:02:18.558970 2926 tablet_service.cc:1467] Processing CreateTablet for tablet 13a6de508dd443eb8ac00ba0c8383761 (DEFAULT_TABLE table=test-table [id=2177f52914114ef39b4d0de4506eb825]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:02:18.559360 3060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13a6de508dd443eb8ac00ba0c8383761. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:02:18.559360 2926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13a6de508dd443eb8ac00ba0c8383761. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:02:18.559275 3194 tablet_service.cc:1467] Processing CreateTablet for tablet 13a6de508dd443eb8ac00ba0c8383761 (DEFAULT_TABLE table=test-table [id=2177f52914114ef39b4d0de4506eb825]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:02:18.559597 3194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13a6de508dd443eb8ac00ba0c8383761. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:02:18.562296 3274 tablet_bootstrap.cc:492] T 13a6de508dd443eb8ac00ba0c8383761 P 7a88cd1f442d4fe4bdae9d7fb8c45823: Bootstrap starting.
I20260927 04:02:18.562564 3275 tablet_bootstrap.cc:492] T 13a6de508dd443eb8ac00ba0c8383761 P 1c979348306b4c0cbcf7785428e860b9: Bootstrap starting.
I20260927 04:02:18.562808 3276 tablet_bootstrap.cc:492] T 13a6de508dd443eb8ac00ba0c8383761 P 4680d95bd48b4a57a6794c6a42d9ca68: Bootstrap starting.
I20260927 04:02:18.563019 3274 tablet_bootstrap.cc:654] T 13a6de508dd443eb8ac00ba0c8383761 P 7a88cd1f442d4fe4bdae9d7fb8c45823: Neither blocks nor log segments found. Creating new log.
I20260927 04:02:18.563508 3276 tablet_bootstrap.cc:654] T 13a6de508dd443eb8ac00ba0c8383761 P 4680d95bd48b4a57a6794c6a42d9ca68: Neither blocks nor log segments found. Creating new log.
I20260927 04:02:18.563650 3274 log.cc:824] T 13a6de508dd443eb8ac00ba0c8383761 P 7a88cd1f442d4fe4bdae9d7fb8c45823: Log is configured to *not* fsync() on all Append() calls
I20260927 04:02:18.564132 3276 log.cc:824] T 13a6de508dd443eb8ac00ba0c8383761 P 4680d95bd48b4a57a6794c6a42d9ca68: Log is configured to *not* fsync() on all Append() calls
I20260927 04:02:18.564450 3274 tablet_bootstrap.cc:492] T 13a6de508dd443eb8ac00ba0c8383761 P 7a88cd1f442d4fe4bdae9d7fb8c45823: No bootstrap required, opened a new log
I20260927 04:02:18.564523 3274 ts_tablet_manager.cc:1397] T 13a6de508dd443eb8ac00ba0c8383761 P 7a88cd1f442d4fe4bdae9d7fb8c45823: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 04:02:18.564826 3276 tablet_bootstrap.cc:492] T 13a6de508dd443eb8ac00ba0c8383761 P 4680d95bd48b4a57a6794c6a42d9ca68: No bootstrap required, opened a new log
I20260927 04:02:18.564950 3276 ts_tablet_manager.cc:1397] T 13a6de508dd443eb8ac00ba0c8383761 P 4680d95bd48b4a57a6794c6a42d9ca68: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 04:02:18.564972 3275 tablet_bootstrap.cc:654] T 13a6de508dd443eb8ac00ba0c8383761 P 1c979348306b4c0cbcf7785428e860b9: Neither blocks nor log segments found. Creating new log.
I20260927 04:02:18.565534 3275 log.cc:824] T 13a6de508dd443eb8ac00ba0c8383761 P 1c979348306b4c0cbcf7785428e860b9: Log is configured to *not* fsync() on all Append() calls
I20260927 04:02:18.566363 3275 tablet_bootstrap.cc:492] T 13a6de508dd443eb8ac00ba0c8383761 P 1c979348306b4c0cbcf7785428e860b9: No bootstrap required, opened a new log
I20260927 04:02:18.566478 3275 ts_tablet_manager.cc:1397] T 13a6de508dd443eb8ac00ba0c8383761 P 1c979348306b4c0cbcf7785428e860b9: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 04:02:18.566524 3276 raft_consensus.cc:348] T 13a6de508dd443eb8ac00ba0c8383761 P 4680d95bd48b4a57a6794c6a42d9ca68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c979348306b4c0cbcf7785428e860b9" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 36963 } } peers { permanent_uuid: "4680d95bd48b4a57a6794c6a42d9ca68" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 40297 } } peers { permanent_uuid: "7a88cd1f442d4fe4bdae9d7fb8c45823" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 43347 } }
I20260927 04:02:18.566690 3276 raft_consensus.cc:374] T 13a6de508dd443eb8ac00ba0c8383761 P 4680d95bd48b4a57a6794c6a42d9ca68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:02:18.566761 3276 raft_consensus.cc:729] T 13a6de508dd443eb8ac00ba0c8383761 P 4680d95bd48b4a57a6794c6a42d9ca68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4680d95bd48b4a57a6794c6a42d9ca68, State: Initialized, Role: FOLLOWER
I20260927 04:02:18.566915 3276 consensus_queue.cc:260] T 13a6de508dd443eb8ac00ba0c8383761 P 4680d95bd48b4a57a6794c6a42d9ca68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c979348306b4c0cbcf7785428e860b9" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 36963 } } peers { permanent_uuid: "4680d95bd48b4a57a6794c6a42d9ca68" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 40297 } } peers { permanent_uuid: "7a88cd1f442d4fe4bdae9d7fb8c45823" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 43347 } }
I20260927 04:02:18.566830 3274 raft_consensus.cc:348] T 13a6de508dd443eb8ac00ba0c8383761 P 7a88cd1f442d4fe4bdae9d7fb8c45823 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c979348306b4c0cbcf7785428e860b9" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 36963 } } peers { permanent_uuid: "4680d95bd48b4a57a6794c6a42d9ca68" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 40297 } } peers { permanent_uuid: "7a88cd1f442d4fe4bdae9d7fb8c45823" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 43347 } }
I20260927 04:02:18.567013 3274 raft_consensus.cc:374] T 13a6de508dd443eb8ac00ba0c8383761 P 7a88cd1f442d4fe4bdae9d7fb8c45823 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:02:18.567102 3274 raft_consensus.cc:729] T 13a6de508dd443eb8ac00ba0c8383761 P 7a88cd1f442d4fe4bdae9d7fb8c45823 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a88cd1f442d4fe4bdae9d7fb8c45823, State: Initialized, Role: FOLLOWER
I20260927 04:02:18.567224 3276 ts_tablet_manager.cc:1428] T 13a6de508dd443eb8ac00ba0c8383761 P 4680d95bd48b4a57a6794c6a42d9ca68: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 04:02:18.567276 3274 consensus_queue.cc:260] T 13a6de508dd443eb8ac00ba0c8383761 P 7a88cd1f442d4fe4bdae9d7fb8c45823 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c979348306b4c0cbcf7785428e860b9" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 36963 } } peers { permanent_uuid: "4680d95bd48b4a57a6794c6a42d9ca68" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 40297 } } peers { permanent_uuid: "7a88cd1f442d4fe4bdae9d7fb8c45823" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 43347 } }
I20260927 04:02:18.567414 2992 heartbeater.cc:503] Master 127.15.185.126:33703 was elected leader, sending a full tablet report...
I20260927 04:02:18.567577 3260 heartbeater.cc:503] Master 127.15.185.126:33703 was elected leader, sending a full tablet report...
I20260927 04:02:18.567797 3274 ts_tablet_manager.cc:1428] T 13a6de508dd443eb8ac00ba0c8383761 P 7a88cd1f442d4fe4bdae9d7fb8c45823: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 04:02:18.568650 3275 raft_consensus.cc:348] T 13a6de508dd443eb8ac00ba0c8383761 P 1c979348306b4c0cbcf7785428e860b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c979348306b4c0cbcf7785428e860b9" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 36963 } } peers { permanent_uuid: "4680d95bd48b4a57a6794c6a42d9ca68" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 40297 } } peers { permanent_uuid: "7a88cd1f442d4fe4bdae9d7fb8c45823" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 43347 } }
I20260927 04:02:18.568837 3275 raft_consensus.cc:374] T 13a6de508dd443eb8ac00ba0c8383761 P 1c979348306b4c0cbcf7785428e860b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:02:18.568886 3275 raft_consensus.cc:729] T 13a6de508dd443eb8ac00ba0c8383761 P 1c979348306b4c0cbcf7785428e860b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c979348306b4c0cbcf7785428e860b9, State: Initialized, Role: FOLLOWER
I20260927 04:02:18.568994 3275 consensus_queue.cc:260] T 13a6de508dd443eb8ac00ba0c8383761 P 1c979348306b4c0cbcf7785428e860b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c979348306b4c0cbcf7785428e860b9" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 36963 } } peers { permanent_uuid: "4680d95bd48b4a57a6794c6a42d9ca68" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 40297 } } peers { permanent_uuid: "7a88cd1f442d4fe4bdae9d7fb8c45823" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 43347 } }
I20260927 04:02:18.569281 3275 ts_tablet_manager.cc:1428] T 13a6de508dd443eb8ac00ba0c8383761 P 1c979348306b4c0cbcf7785428e860b9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 04:02:18.569299 3126 heartbeater.cc:503] Master 127.15.185.126:33703 was elected leader, sending a full tablet report...
W20260927 04:02:18.609622 3127 tablet.cc:2369] T 13a6de508dd443eb8ac00ba0c8383761 P 1c979348306b4c0cbcf7785428e860b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 04:02:18.694855 3281 raft_consensus.cc:482] T 13a6de508dd443eb8ac00ba0c8383761 P 7a88cd1f442d4fe4bdae9d7fb8c45823 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:02:18.694986 3281 raft_consensus.cc:504] T 13a6de508dd443eb8ac00ba0c8383761 P 7a88cd1f442d4fe4bdae9d7fb8c45823 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c979348306b4c0cbcf7785428e860b9" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 36963 } } peers { permanent_uuid: "4680d95bd48b4a57a6794c6a42d9ca68" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 40297 } } peers { permanent_uuid: "7a88cd1f442d4fe4bdae9d7fb8c45823" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 43347 } }
I20260927 04:02:18.695410 3281 leader_election.cc:288] T 13a6de508dd443eb8ac00ba0c8383761 P 7a88cd1f442d4fe4bdae9d7fb8c45823 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c979348306b4c0cbcf7785428e860b9 (127.15.185.66:36963), 4680d95bd48b4a57a6794c6a42d9ca68 (127.15.185.65:40297)
I20260927 04:02:18.698819 3262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:18.695634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.66:36963 (local address 127.15.185.67:60715)
0927 04:02:18.695710 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:18.695743 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:18.695817 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:18.695965 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:18.695966 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:18.695968 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:02:18.696040 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:18.696046 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:18.696716 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:18.696717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:18.697911 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:18.697914 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:18.698549 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:18.698550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:18.698632 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:18.698636 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 04:02:18.698706 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 04:02:18.698747 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 04:02:18.698819 3270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:18.695698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:60715 (local address 127.15.185.66:36963)
0927 04:02:18.695761 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:18.695763 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:18.695798 (+ 35us) server_negotiation.cc:407] Connection header received
0927 04:02:18.695890 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:18.695892 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:18.695904 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:18.695957 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:02:18.696110 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:18.696646 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:18.698007 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:18.698471 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:18.698555 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:18.698564 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:18.698717 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 04:02:18.698754 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 04:02:18.699105 3273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:18.695841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:46005 (local address 127.15.185.65:40297)
0927 04:02:18.695888 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:18.695890 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:18.696042 (+ 152us) server_negotiation.cc:407] Connection header received
0927 04:02:18.696116 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:18.696118 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:18.696130 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:18.696195 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:02:18.696354 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:18.696910 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:18.698233 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:18.698644 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:18.698760 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:18.698769 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:18.698957 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 04:02:18.699008 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 04:02:18.699242 3080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13a6de508dd443eb8ac00ba0c8383761" candidate_uuid: "7a88cd1f442d4fe4bdae9d7fb8c45823" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c979348306b4c0cbcf7785428e860b9" is_pre_election: true
I20260927 04:02:18.699338 3283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:18.695771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.65:40297 (local address 127.15.185.67:46005)
0927 04:02:18.695956 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:18.695966 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:18.696054 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:18.696198 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:18.696199 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:18.696201 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:02:18.696272 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:18.696274 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:18.697030 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:18.697031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:18.698130 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:18.698133 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:18.698764 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:18.698765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:18.698847 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:18.698850 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 04:02:18.699216 (+ 366us) client_negotiation.cc:240] Negotiation successful
0927 04:02:18.699262 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":115,"threads_started":1}
I20260927 04:02:18.699433 3080 raft_consensus.cc:2454] T 13a6de508dd443eb8ac00ba0c8383761 P 1c979348306b4c0cbcf7785428e860b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a88cd1f442d4fe4bdae9d7fb8c45823 in term 0.
I20260927 04:02:18.699692 3147 leader_election.cc:302] T 13a6de508dd443eb8ac00ba0c8383761 P 7a88cd1f442d4fe4bdae9d7fb8c45823 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c979348306b4c0cbcf7785428e860b9, 7a88cd1f442d4fe4bdae9d7fb8c45823; no voters:
I20260927 04:02:18.699810 2946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13a6de508dd443eb8ac00ba0c8383761" candidate_uuid: "7a88cd1f442d4fe4bdae9d7fb8c45823" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4680d95bd48b4a57a6794c6a42d9ca68" is_pre_election: true
I20260927 04:02:18.699951 2946 raft_consensus.cc:2454] T 13a6de508dd443eb8ac00ba0c8383761 P 4680d95bd48b4a57a6794c6a42d9ca68 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a88cd1f442d4fe4bdae9d7fb8c45823 in term 0.
I20260927 04:02:18.700033 3281 raft_consensus.cc:2781] T 13a6de508dd443eb8ac00ba0c8383761 P 7a88cd1f442d4fe4bdae9d7fb8c45823 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 04:02:18.700115 3281 raft_consensus.cc:482] T 13a6de508dd443eb8ac00ba0c8383761 P 7a88cd1f442d4fe4bdae9d7fb8c45823 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 04:02:18.700140 3281 raft_consensus.cc:3037] T 13a6de508dd443eb8ac00ba0c8383761 P 7a88cd1f442d4fe4bdae9d7fb8c45823 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:02:18.700842 3281 raft_consensus.cc:504] T 13a6de508dd443eb8ac00ba0c8383761 P 7a88cd1f442d4fe4bdae9d7fb8c45823 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c979348306b4c0cbcf7785428e860b9" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 36963 } } peers { permanent_uuid: "4680d95bd48b4a57a6794c6a42d9ca68" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 40297 } } peers { permanent_uuid: "7a88cd1f442d4fe4bdae9d7fb8c45823" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 43347 } }
I20260927 04:02:18.700980 3281 leader_election.cc:288] T 13a6de508dd443eb8ac00ba0c8383761 P 7a88cd1f442d4fe4bdae9d7fb8c45823 [CANDIDATE]: Term 1 election: Requested vote from peers 1c979348306b4c0cbcf7785428e860b9 (127.15.185.66:36963), 4680d95bd48b4a57a6794c6a42d9ca68 (127.15.185.65:40297)
I20260927 04:02:18.701154 2946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13a6de508dd443eb8ac00ba0c8383761" candidate_uuid: "7a88cd1f442d4fe4bdae9d7fb8c45823" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4680d95bd48b4a57a6794c6a42d9ca68"
I20260927 04:02:18.701152 3080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13a6de508dd443eb8ac00ba0c8383761" candidate_uuid: "7a88cd1f442d4fe4bdae9d7fb8c45823" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c979348306b4c0cbcf7785428e860b9"
I20260927 04:02:18.701236 2946 raft_consensus.cc:3037] T 13a6de508dd443eb8ac00ba0c8383761 P 4680d95bd48b4a57a6794c6a42d9ca68 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:02:18.701236 3080 raft_consensus.cc:3037] T 13a6de508dd443eb8ac00ba0c8383761 P 1c979348306b4c0cbcf7785428e860b9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:02:18.702124 2946 raft_consensus.cc:2454] T 13a6de508dd443eb8ac00ba0c8383761 P 4680d95bd48b4a57a6794c6a42d9ca68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a88cd1f442d4fe4bdae9d7fb8c45823 in term 1.
I20260927 04:02:18.702124 3080 raft_consensus.cc:2454] T 13a6de508dd443eb8ac00ba0c8383761 P 1c979348306b4c0cbcf7785428e860b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a88cd1f442d4fe4bdae9d7fb8c45823 in term 1.
I20260927 04:02:18.702313 3147 leader_election.cc:302] T 13a6de508dd443eb8ac00ba0c8383761 P 7a88cd1f442d4fe4bdae9d7fb8c45823 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4680d95bd48b4a57a6794c6a42d9ca68, 7a88cd1f442d4fe4bdae9d7fb8c45823; no voters:
I20260927 04:02:18.702402 3281 raft_consensus.cc:2781] T 13a6de508dd443eb8ac00ba0c8383761 P 7a88cd1f442d4fe4bdae9d7fb8c45823 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:02:18.702544 3281 raft_consensus.cc:686] T 13a6de508dd443eb8ac00ba0c8383761 P 7a88cd1f442d4fe4bdae9d7fb8c45823 [term 1 LEADER]: Becoming Leader. State: Replica: 7a88cd1f442d4fe4bdae9d7fb8c45823, State: Running, Role: LEADER
I20260927 04:02:18.702617 3281 consensus_queue.cc:237] T 13a6de508dd443eb8ac00ba0c8383761 P 7a88cd1f442d4fe4bdae9d7fb8c45823 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c979348306b4c0cbcf7785428e860b9" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 36963 } } peers { permanent_uuid: "4680d95bd48b4a57a6794c6a42d9ca68" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 40297 } } peers { permanent_uuid: "7a88cd1f442d4fe4bdae9d7fb8c45823" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 43347 } }
I20260927 04:02:18.703507 2798 catalog_manager.cc:5495] T 13a6de508dd443eb8ac00ba0c8383761 P 7a88cd1f442d4fe4bdae9d7fb8c45823 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a88cd1f442d4fe4bdae9d7fb8c45823 (127.15.185.67). New cstate: current_term: 1 leader_uuid: "7a88cd1f442d4fe4bdae9d7fb8c45823" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c979348306b4c0cbcf7785428e860b9" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 36963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4680d95bd48b4a57a6794c6a42d9ca68" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 40297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a88cd1f442d4fe4bdae9d7fb8c45823" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 43347 } health_report { overall_health: HEALTHY } } }
W20260927 04:02:18.713335 2993 tablet.cc:2369] T 13a6de508dd443eb8ac00ba0c8383761 P 4680d95bd48b4a57a6794c6a42d9ca68: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 04:02:18.738344 3271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:18.735856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34298 (local address 127.15.185.67:43347)
0927 04:02:18.735904 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:18.735906 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:18.735932 (+ 26us) server_negotiation.cc:407] Connection header received
0927 04:02:18.735974 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:18.735975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:18.735996 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:18.736079 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:02:18.736230 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:18.736704 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:18.737511 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:18.737724 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:18.737792 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:18.737968 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:02:18.738089 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:02:18.738193 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:18.738210 (+ 17us) server_negotiation.cc:299] Negotiation successful
0927 04:02:18.738232 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 04:02:18.740304 2946 raft_consensus.cc:1261] T 13a6de508dd443eb8ac00ba0c8383761 P 4680d95bd48b4a57a6794c6a42d9ca68 [term 1 FOLLOWER]: Refusing update from remote peer 7a88cd1f442d4fe4bdae9d7fb8c45823: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:02:18.740316 3080 raft_consensus.cc:1261] T 13a6de508dd443eb8ac00ba0c8383761 P 1c979348306b4c0cbcf7785428e860b9 [term 1 FOLLOWER]: Refusing update from remote peer 7a88cd1f442d4fe4bdae9d7fb8c45823: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:02:18.740643 3281 consensus_queue.cc:1035] T 13a6de508dd443eb8ac00ba0c8383761 P 7a88cd1f442d4fe4bdae9d7fb8c45823 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4680d95bd48b4a57a6794c6a42d9ca68" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 40297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:02:18.740799 3284 consensus_queue.cc:1035] T 13a6de508dd443eb8ac00ba0c8383761 P 7a88cd1f442d4fe4bdae9d7fb8c45823 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c979348306b4c0cbcf7785428e860b9" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 36963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:02:18.743021 3287 mvcc.cc:205] Tried to move back new op lower bound from 7333813201877438464 to 7333813201726656512. Current Snapshot: MvccSnapshot[applied={T|T < 7333813201877438464}]
I20260927 04:02:18.743494 3289 mvcc.cc:205] Tried to move back new op lower bound from 7333813201877438464 to 7333813201726656512. Current Snapshot: MvccSnapshot[applied={T|T < 7333813201877438464}]
I20260927 04:02:18.743654 3288 mvcc.cc:205] Tried to move back new op lower bound from 7333813201877438464 to 7333813201726656512. Current Snapshot: MvccSnapshot[applied={T|T < 7333813201877438464}]
I20260927 04:02:18.745590 2799 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:02:18.745669 2799 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:02:18.746353 2799 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a12db7fe70944faaad5ac93894d1b3ea: Sending DeleteTablet for 3 replicas of tablet 13a6de508dd443eb8ac00ba0c8383761
I20260927 04:02:18.746675 3194 tablet_service.cc:1514] Processing DeleteTablet for tablet 13a6de508dd443eb8ac00ba0c8383761 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:02:18 UTC) from {username='kudu', principal='kudu/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:34286
I20260927 04:02:18.746694 2926 tablet_service.cc:1514] Processing DeleteTablet for tablet 13a6de508dd443eb8ac00ba0c8383761 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:02:18 UTC) from {username='kudu', principal='kudu/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:50112
I20260927 04:02:18.746716 3060 tablet_service.cc:1514] Processing DeleteTablet for tablet 13a6de508dd443eb8ac00ba0c8383761 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:02:18 UTC) from {username='kudu', principal='kudu/127.15.185.126@KRBTEST.COM'} at 127.0.0.1:49090
I20260927 04:02:18.746897 3297 tablet_replica.cc:330] T 13a6de508dd443eb8ac00ba0c8383761 P 7a88cd1f442d4fe4bdae9d7fb8c45823: stopping tablet replica
I20260927 04:02:18.746994 3298 tablet_replica.cc:330] T 13a6de508dd443eb8ac00ba0c8383761 P 4680d95bd48b4a57a6794c6a42d9ca68: stopping tablet replica
I20260927 04:02:18.747113 3298 raft_consensus.cc:2229] T 13a6de508dd443eb8ac00ba0c8383761 P 4680d95bd48b4a57a6794c6a42d9ca68 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:02:18.747216 3298 raft_consensus.cc:2258] T 13a6de508dd443eb8ac00ba0c8383761 P 4680d95bd48b4a57a6794c6a42d9ca68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:02:18.747370 3297 raft_consensus.cc:2229] T 13a6de508dd443eb8ac00ba0c8383761 P 7a88cd1f442d4fe4bdae9d7fb8c45823 [term 1 LEADER]: Raft consensus shutting down.
I20260927 04:02:18.747392 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 2862
I20260927 04:02:18.748013 3297 raft_consensus.cc:2258] T 13a6de508dd443eb8ac00ba0c8383761 P 7a88cd1f442d4fe4bdae9d7fb8c45823 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:02:18.748432 3297 ts_tablet_manager.cc:1905] T 13a6de508dd443eb8ac00ba0c8383761 P 7a88cd1f442d4fe4bdae9d7fb8c45823: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:02:18.750283 3297 ts_tablet_manager.cc:1918] T 13a6de508dd443eb8ac00ba0c8383761 P 7a88cd1f442d4fe4bdae9d7fb8c45823: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:02:18.750342 3297 log.cc:1196] T 13a6de508dd443eb8ac00ba0c8383761 P 7a88cd1f442d4fe4bdae9d7fb8c45823: Deleting WAL directory at /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/ts-2/wal/wals/13a6de508dd443eb8ac00ba0c8383761
I20260927 04:02:18.750593 3297 ts_tablet_manager.cc:1939] T 13a6de508dd443eb8ac00ba0c8383761 P 7a88cd1f442d4fe4bdae9d7fb8c45823: Deleting consensus metadata
I20260927 04:02:18.751010 2786 catalog_manager.cc:4841] TS 7a88cd1f442d4fe4bdae9d7fb8c45823 (127.15.185.67:43347): tablet 13a6de508dd443eb8ac00ba0c8383761 (table test-table [id=2177f52914114ef39b4d0de4506eb825]) successfully deleted
I20260927 04:02:18.751160 3299 tablet_replica.cc:330] T 13a6de508dd443eb8ac00ba0c8383761 P 1c979348306b4c0cbcf7785428e860b9: stopping tablet replica
I20260927 04:02:18.751281 3299 raft_consensus.cc:2229] T 13a6de508dd443eb8ac00ba0c8383761 P 1c979348306b4c0cbcf7785428e860b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:02:18.751396 3299 raft_consensus.cc:2258] T 13a6de508dd443eb8ac00ba0c8383761 P 1c979348306b4c0cbcf7785428e860b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:02:18.751772 3299 ts_tablet_manager.cc:1905] T 13a6de508dd443eb8ac00ba0c8383761 P 1c979348306b4c0cbcf7785428e860b9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 04:02:18.753167 2783 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.185.65:40297 (error 108)
I20260927 04:02:18.753201 3299 ts_tablet_manager.cc:1918] T 13a6de508dd443eb8ac00ba0c8383761 P 1c979348306b4c0cbcf7785428e860b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:02:18.753254 3299 log.cc:1196] T 13a6de508dd443eb8ac00ba0c8383761 P 1c979348306b4c0cbcf7785428e860b9: Deleting WAL directory at /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481498018407-16101-0/minicluster-data/ts-1/wal/wals/13a6de508dd443eb8ac00ba0c8383761
I20260927 04:02:18.753482 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 2997
I20260927 04:02:18.753577 3299 ts_tablet_manager.cc:1939] T 13a6de508dd443eb8ac00ba0c8383761 P 1c979348306b4c0cbcf7785428e860b9: Deleting consensus metadata
I20260927 04:02:18.758978 3272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:18.758828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.65:40297 (local address 127.0.0.1:50118)
0927 04:02:18.758887 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:18.758932 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.65:40297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 04:02:18.758997 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 3131
I20260927 04:02:18.759037 3268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:18.758902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.66:36963 (local address 127.0.0.1:49096)
0927 04:02:18.758941 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:18.758968 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.66:36963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260927 04:02:18.760344 2783 catalog_manager.cc:4584] TS 4680d95bd48b4a57a6794c6a42d9ca68 (127.15.185.65:40297): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 13a6de508dd443eb8ac00ba0c8383761: Network error: Client connection negotiation failed: client connection to 127.15.185.65:40297: connect: Connection refused (error 111)
I20260927 04:02:18.763686 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 2767
2026-09-27T04:02:18Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2036 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-27T04:02:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:02:18Z Disabled control of system clock
I20260927 04:02:18.792524 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:34467
--webserver_interface=127.15.185.126
--webserver_port=0
--builtin_ntp_servers=127.15.185.84:40073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.185.126:34467
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 04:02:18.872277 3304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:02:18.872496 3304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:02:18.872519 3304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:02:18.873610 3304 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:02:18.873646 3304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:02:18.873658 3304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:02:18.873670 3304 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:02:18.873682 3304 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:02:18.874975 3304 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:40073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.185.126:34467
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:34467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:02:18.875267 3304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:02:18.875562 3304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:02:18.880630 3312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:18.880734 3304 server_base.cc:1034] running on GCE node
W20260927 04:02:18.880646 3310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:18.880607 3309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:18.881105 3304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:02:18.881359 3304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:02:18.882501 3304 hybrid_clock.cc:648] HybridClock initialized: now 1790481738882488 us; error 28 us; skew 500 ppm
I20260927 04:02:18.883765 3304 webserver.cc:460] Webserver started at http://127.15.185.126:37605/ using document root <none> and password file <none>
I20260927 04:02:18.884002 3304 fs_manager.cc:359] Metadata directory not provided
I20260927 04:02:18.884047 3304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:02:18.884229 3304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:02:18.885313 3304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/master-0/data/instance:
uuid: "10f45f8e57e04904aeb2a97aec4cd8f8"
format_stamp: "Formatted at 2026-09-27 04:02:18 on dist-test-slave-glgz"
server_key: "9a3619c447f80bc7b67f261ccd34ae3a"
server_key_iv: "1d1db58f9914fa789dc602fe82c78b09"
server_key_version: "encryptionkey@0"
I20260927 04:02:18.885665 3304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/master-0/wal/instance:
uuid: "10f45f8e57e04904aeb2a97aec4cd8f8"
format_stamp: "Formatted at 2026-09-27 04:02:18 on dist-test-slave-glgz"
server_key: "9a3619c447f80bc7b67f261ccd34ae3a"
server_key_iv: "1d1db58f9914fa789dc602fe82c78b09"
server_key_version: "encryptionkey@0"
I20260927 04:02:18.886965 3304 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 04:02:18.887796 3318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:18.887982 3304 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 04:02:18.888072 3304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/master-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/master-0/wal
uuid: "10f45f8e57e04904aeb2a97aec4cd8f8"
format_stamp: "Formatted at 2026-09-27 04:02:18 on dist-test-slave-glgz"
server_key: "9a3619c447f80bc7b67f261ccd34ae3a"
server_key_iv: "1d1db58f9914fa789dc602fe82c78b09"
server_key_version: "encryptionkey@0"
I20260927 04:02:18.888147 3304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:02:18.906629 3304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:02:18.906968 3304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:02:18.907148 3304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:02:18.911092 3304 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.126:34467
I20260927 04:02:18.911128 3370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.126:34467 every 8 connection(s)
I20260927 04:02:18.911485 3304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
I20260927 04:02:18.912426 3371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:02:18.914742 3371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10f45f8e57e04904aeb2a97aec4cd8f8: Bootstrap starting.
I20260927 04:02:18.915367 3371 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10f45f8e57e04904aeb2a97aec4cd8f8: Neither blocks nor log segments found. Creating new log.
I20260927 04:02:18.915659 3371 log.cc:824] T 00000000000000000000000000000000 P 10f45f8e57e04904aeb2a97aec4cd8f8: Log is configured to *not* fsync() on all Append() calls
I20260927 04:02:18.916388 3371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10f45f8e57e04904aeb2a97aec4cd8f8: No bootstrap required, opened a new log
I20260927 04:02:18.916564 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 3304
I20260927 04:02:18.916683 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/master-0/wal/instance
I20260927 04:02:18.916911 16101 external_mini_cluster.cc:1428] Setting key b01c33ee6dd221ed9c550c36e71e8410
I20260927 04:02:18.918706 3371 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10f45f8e57e04904aeb2a97aec4cd8f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10f45f8e57e04904aeb2a97aec4cd8f8" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 34467 } }
I20260927 04:02:18.918882 3371 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10f45f8e57e04904aeb2a97aec4cd8f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:02:18.918921 3371 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10f45f8e57e04904aeb2a97aec4cd8f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f45f8e57e04904aeb2a97aec4cd8f8, State: Initialized, Role: FOLLOWER
I20260927 04:02:18.919106 3371 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10f45f8e57e04904aeb2a97aec4cd8f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10f45f8e57e04904aeb2a97aec4cd8f8" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 34467 } }
I20260927 04:02:18.919198 3371 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10f45f8e57e04904aeb2a97aec4cd8f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:02:18.919230 3371 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10f45f8e57e04904aeb2a97aec4cd8f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:02:18.919282 3371 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10f45f8e57e04904aeb2a97aec4cd8f8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:02:18.920019 3371 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10f45f8e57e04904aeb2a97aec4cd8f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10f45f8e57e04904aeb2a97aec4cd8f8" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 34467 } }
I20260927 04:02:18.920123 3371 leader_election.cc:302] T 00000000000000000000000000000000 P 10f45f8e57e04904aeb2a97aec4cd8f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10f45f8e57e04904aeb2a97aec4cd8f8; no voters:
I20260927 04:02:18.920274 3371 leader_election.cc:288] T 00000000000000000000000000000000 P 10f45f8e57e04904aeb2a97aec4cd8f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:02:18.920331 3376 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10f45f8e57e04904aeb2a97aec4cd8f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:02:18.920500 3376 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10f45f8e57e04904aeb2a97aec4cd8f8 [term 1 LEADER]: Becoming Leader. State: Replica: 10f45f8e57e04904aeb2a97aec4cd8f8, State: Running, Role: LEADER
I20260927 04:02:18.920511 3371 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10f45f8e57e04904aeb2a97aec4cd8f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:02:18.920614 3376 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10f45f8e57e04904aeb2a97aec4cd8f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10f45f8e57e04904aeb2a97aec4cd8f8" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 34467 } }
I20260927 04:02:18.922204 3378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10f45f8e57e04904aeb2a97aec4cd8f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10f45f8e57e04904aeb2a97aec4cd8f8. Latest consensus state: current_term: 1 leader_uuid: "10f45f8e57e04904aeb2a97aec4cd8f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10f45f8e57e04904aeb2a97aec4cd8f8" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 34467 } } }
I20260927 04:02:18.922204 3377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10f45f8e57e04904aeb2a97aec4cd8f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "10f45f8e57e04904aeb2a97aec4cd8f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10f45f8e57e04904aeb2a97aec4cd8f8" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 34467 } } }
I20260927 04:02:18.922569 3377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10f45f8e57e04904aeb2a97aec4cd8f8 [sys.catalog]: This master's current role is: LEADER
I20260927 04:02:18.922318 3378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10f45f8e57e04904aeb2a97aec4cd8f8 [sys.catalog]: This master's current role is: LEADER
W20260927 04:02:18.922875 3391 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 10f45f8e57e04904aeb2a97aec4cd8f8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 04:02:18.922935 3391 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 04:02:18.923026 3392 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:02:18.923221 3392 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:02:18.924357 3375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:18.917445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37722 (local address 127.15.185.126:34467)
0927 04:02:18.917661 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:18.917671 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:18.917705 (+ 34us) server_negotiation.cc:407] Connection header received
0927 04:02:18.918649 (+ 944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:18.918660 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:18.918741 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:18.918894 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:18.919148 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:18.920871 (+ 1723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:18.921903 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:18.922565 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:18.922690 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:18.923621 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:18.923625 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:18.923625 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:02:18.923839 (+ 214us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:18.923881 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:02:18.923897 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:18.923984 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:18.924063 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 04:02:18.924143 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":88,"threads_started":1}
I20260927 04:02:18.925099 3392 catalog_manager.cc:1347] Generated new cluster ID: 8af6ca65e0ee435ab155237b44749d8c
I20260927 04:02:18.925149 3392 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:02:18.956135 3392 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:02:18.956921 3392 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:02:18.976128 3392 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 10f45f8e57e04904aeb2a97aec4cd8f8: Generated new TSK 0
I20260927 04:02:18.976374 3392 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 04:02:18.981956 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.65:0
--local_ip_for_outbound_sockets=127.15.185.65
--webserver_interface=127.15.185.65
--webserver_port=0
--tserver_master_addrs=127.15.185.126:34467
--builtin_ntp_servers=127.15.185.84:40073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 04:02:19.064816 3395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:02:19.065019 3395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:02:19.065047 3395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:02:19.066134 3395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:02:19.066169 3395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:02:19.066193 3395 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.65
I20260927 04:02:19.067662 3395 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:40073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.185.65
--webserver_port=0
--tserver_master_addrs=127.15.185.126:34467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3395
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.65
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:02:19.067867 3395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:02:19.068125 3395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:02:19.073143 3403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:19.073129 3400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:19.073112 3401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:19.073403 3395 server_base.cc:1034] running on GCE node
I20260927 04:02:19.073634 3395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:02:19.074021 3395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:02:19.075204 3395 hybrid_clock.cc:648] HybridClock initialized: now 1790481739075174 us; error 48 us; skew 500 ppm
I20260927 04:02:19.076481 3395 webserver.cc:460] Webserver started at http://127.15.185.65:40391/ using document root <none> and password file <none>
I20260927 04:02:19.076728 3395 fs_manager.cc:359] Metadata directory not provided
I20260927 04:02:19.076792 3395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:02:19.076946 3395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:02:19.078015 3395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/ts-0/data/instance:
uuid: "44aeff1d421c48149fae844365c5754e"
format_stamp: "Formatted at 2026-09-27 04:02:19 on dist-test-slave-glgz"
server_key: "8e088f95cb0ff4ab09768a97fdafecc5"
server_key_iv: "1c8da4214e5e032cc0c81f23c028fd2b"
server_key_version: "encryptionkey@0"
I20260927 04:02:19.078362 3395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/ts-0/wal/instance:
uuid: "44aeff1d421c48149fae844365c5754e"
format_stamp: "Formatted at 2026-09-27 04:02:19 on dist-test-slave-glgz"
server_key: "8e088f95cb0ff4ab09768a97fdafecc5"
server_key_iv: "1c8da4214e5e032cc0c81f23c028fd2b"
server_key_version: "encryptionkey@0"
I20260927 04:02:19.079769 3395 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 04:02:19.080520 3409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:19.080714 3395 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 04:02:19.080809 3395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/ts-0/wal
uuid: "44aeff1d421c48149fae844365c5754e"
format_stamp: "Formatted at 2026-09-27 04:02:19 on dist-test-slave-glgz"
server_key: "8e088f95cb0ff4ab09768a97fdafecc5"
server_key_iv: "1c8da4214e5e032cc0c81f23c028fd2b"
server_key_version: "encryptionkey@0"
I20260927 04:02:19.080880 3395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:02:19.096571 3395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:02:19.096921 3395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:02:19.097074 3395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:02:19.097307 3395 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:02:19.097643 3395 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:02:19.097693 3395 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:19.097738 3395 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:02:19.097764 3395 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:19.103415 3395 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.65:33505
I20260927 04:02:19.103494 3522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.65:33505 every 8 connection(s)
I20260927 04:02:19.103834 3395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/ts-0/data/info.pb
I20260927 04:02:19.107355 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 3395
I20260927 04:02:19.107478 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/ts-0/wal/instance
I20260927 04:02:19.107710 16101 external_mini_cluster.cc:1428] Setting key a422a5bfe125de81235ca0bdd785c6ef
I20260927 04:02:19.108827 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.66:0
--local_ip_for_outbound_sockets=127.15.185.66
--webserver_interface=127.15.185.66
--webserver_port=0
--tserver_master_addrs=127.15.185.126:34467
--builtin_ntp_servers=127.15.185.84:40073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 04:02:19.109561 3375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:19.104622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.65:40101 (local address 127.15.185.126:34467)
0927 04:02:19.104687 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:19.104689 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:19.104941 (+ 252us) server_negotiation.cc:407] Connection header received
0927 04:02:19.105417 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:19.105418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:19.105429 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:19.105484 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:19.106152 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:19.106643 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:19.107816 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:19.108066 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:19.108127 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:19.109256 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:19.109258 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:19.109258 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:02:19.109264 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:19.109282 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:02:19.109287 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:19.109346 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:19.109413 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 04:02:19.109442 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 04:02:19.109697 3525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:19.104616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34467 (local address 127.15.185.65:40101)
0927 04:02:19.104809 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:19.104844 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:19.105325 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:19.105537 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:19.105540 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:19.105548 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:02:19.106077 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:19.106082 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:19.106721 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:19.106722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:19.107700 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:19.107703 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:19.108149 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:19.108150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:19.108326 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:19.109089 (+ 763us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 04:02:19.109089 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:02:19.109120 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 04:02:19.109143 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:02:19.109352 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:02:19.109356 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:02:19.109358 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 04:02:19.109409 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 04:02:19.109492 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":101,"threads_started":1}
I20260927 04:02:19.110720 3523 heartbeater.cc:347] Connected to a master server at 127.15.185.126:34467
I20260927 04:02:19.110831 3523 heartbeater.cc:464] Registering TS with master...
I20260927 04:02:19.111011 3523 heartbeater.cc:511] Master 127.15.185.126:34467 requested a full tablet report, sending...
I20260927 04:02:19.111553 3335 ts_manager.cc:194] Registered new tserver with Master: 44aeff1d421c48149fae844365c5754e (127.15.185.65:33505)
I20260927 04:02:19.112167 3335 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.185.65:40101
W20260927 04:02:19.195062 3526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:02:19.195312 3526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:02:19.195349 3526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:02:19.196503 3526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:02:19.196578 3526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:02:19.196621 3526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.66
I20260927 04:02:19.197984 3526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:40073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.185.66
--webserver_port=0
--tserver_master_addrs=127.15.185.126:34467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.66
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:02:19.198267 3526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:02:19.198575 3526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:02:19.203887 3531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:19.204074 3526 server_base.cc:1034] running on GCE node
W20260927 04:02:19.203887 3532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:19.203905 3534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:19.204370 3526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:02:19.204627 3526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:02:19.205816 3526 hybrid_clock.cc:648] HybridClock initialized: now 1790481739205807 us; error 33 us; skew 500 ppm
I20260927 04:02:19.207089 3526 webserver.cc:460] Webserver started at http://127.15.185.66:45459/ using document root <none> and password file <none>
I20260927 04:02:19.207369 3526 fs_manager.cc:359] Metadata directory not provided
I20260927 04:02:19.207435 3526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:02:19.207577 3526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:02:19.208660 3526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/ts-1/data/instance:
uuid: "e7fed43d755c4e3697bf0ad721a80103"
format_stamp: "Formatted at 2026-09-27 04:02:19 on dist-test-slave-glgz"
server_key: "b0a05af7de4eaa8387e82894e2351ce1"
server_key_iv: "cdba830e2ba327061522ea472bb2eb00"
server_key_version: "encryptionkey@0"
I20260927 04:02:19.209019 3526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/ts-1/wal/instance:
uuid: "e7fed43d755c4e3697bf0ad721a80103"
format_stamp: "Formatted at 2026-09-27 04:02:19 on dist-test-slave-glgz"
server_key: "b0a05af7de4eaa8387e82894e2351ce1"
server_key_iv: "cdba830e2ba327061522ea472bb2eb00"
server_key_version: "encryptionkey@0"
I20260927 04:02:19.210371 3526 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 04:02:19.211118 3540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:19.211280 3526 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 04:02:19.211382 3526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/ts-1/wal
uuid: "e7fed43d755c4e3697bf0ad721a80103"
format_stamp: "Formatted at 2026-09-27 04:02:19 on dist-test-slave-glgz"
server_key: "b0a05af7de4eaa8387e82894e2351ce1"
server_key_iv: "cdba830e2ba327061522ea472bb2eb00"
server_key_version: "encryptionkey@0"
I20260927 04:02:19.211459 3526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:02:19.230518 3526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:02:19.230863 3526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:02:19.231010 3526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:02:19.231276 3526 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:02:19.231600 3526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:02:19.231652 3526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:19.231685 3526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:02:19.231726 3526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:19.237627 3526 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.66:44501
I20260927 04:02:19.237680 3653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.66:44501 every 8 connection(s)
I20260927 04:02:19.238013 3526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/ts-1/data/info.pb
I20260927 04:02:19.243382 3375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:19.238730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:40577 (local address 127.15.185.126:34467)
0927 04:02:19.238782 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:19.238784 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:19.239065 (+ 281us) server_negotiation.cc:407] Connection header received
0927 04:02:19.239606 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:19.239607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:19.239616 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:19.239682 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:19.240362 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:19.240853 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:19.241899 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:19.242084 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:19.242141 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:19.243103 (+ 962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:19.243105 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:19.243105 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:02:19.243110 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:19.243124 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:02:19.243128 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:19.243181 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:19.243292 (+ 111us) server_negotiation.cc:299] Negotiation successful
0927 04:02:19.243318 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 04:02:19.243541 3656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:19.238717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34467 (local address 127.15.185.66:40577)
0927 04:02:19.238908 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:19.238951 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:19.239513 (+ 562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:19.239741 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:19.239745 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:19.239755 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:02:19.240289 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:19.240294 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:19.240939 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:19.240940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:19.241811 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:19.241815 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:19.242163 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:19.242164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:19.242272 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:19.242875 (+ 603us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 04:02:19.242876 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:02:19.242945 (+ 69us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 04:02:19.242969 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:02:19.243198 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:02:19.243203 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:02:19.243205 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 04:02:19.243259 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 04:02:19.243343 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":92,"threads_started":1}
I20260927 04:02:19.244074 3654 heartbeater.cc:347] Connected to a master server at 127.15.185.126:34467
I20260927 04:02:19.244205 3654 heartbeater.cc:464] Registering TS with master...
I20260927 04:02:19.244345 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 3526
I20260927 04:02:19.244427 3654 heartbeater.cc:511] Master 127.15.185.126:34467 requested a full tablet report, sending...
I20260927 04:02:19.244453 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/ts-1/wal/instance
I20260927 04:02:19.244652 16101 external_mini_cluster.cc:1428] Setting key 9a8a70ddf46480a9adc202bec81f36cb
I20260927 04:02:19.244894 3335 ts_manager.cc:194] Registered new tserver with Master: e7fed43d755c4e3697bf0ad721a80103 (127.15.185.66:44501)
I20260927 04:02:19.245484 3335 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.185.66:40577
I20260927 04:02:19.245731 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.185.67:0
--local_ip_for_outbound_sockets=127.15.185.67
--webserver_interface=127.15.185.67
--webserver_port=0
--tserver_master_addrs=127.15.185.126:34467
--builtin_ntp_servers=127.15.185.84:40073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 04:02:19.327934 3657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:02:19.328148 3657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:02:19.328169 3657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:02:19.329274 3657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:02:19.329315 3657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:02:19.329340 3657 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.185.67
I20260927 04:02:19.331037 3657 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:40073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.185.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.185.67
--webserver_port=0
--tserver_master_addrs=127.15.185.126:34467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.185.67
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:02:19.331300 3657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:02:19.331547 3657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:02:19.336568 3662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:19.336619 3663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:19.336596 3657 server_base.cc:1034] running on GCE node
W20260927 04:02:19.336575 3665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:19.337052 3657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:02:19.337265 3657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:02:19.338398 3657 hybrid_clock.cc:648] HybridClock initialized: now 1790481739338386 us; error 28 us; skew 500 ppm
I20260927 04:02:19.339591 3657 webserver.cc:460] Webserver started at http://127.15.185.67:41659/ using document root <none> and password file <none>
I20260927 04:02:19.339807 3657 fs_manager.cc:359] Metadata directory not provided
I20260927 04:02:19.339850 3657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:02:19.339941 3657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:02:19.340952 3657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/ts-2/data/instance:
uuid: "4f8f299464cb4f8d89a0bbd9a244eb7b"
format_stamp: "Formatted at 2026-09-27 04:02:19 on dist-test-slave-glgz"
server_key: "687ebc91e3b7c3ce1662bfa6f4c13a9d"
server_key_iv: "4edee58051d12c4c83116822e0929cc3"
server_key_version: "encryptionkey@0"
I20260927 04:02:19.341255 3657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/ts-2/wal/instance:
uuid: "4f8f299464cb4f8d89a0bbd9a244eb7b"
format_stamp: "Formatted at 2026-09-27 04:02:19 on dist-test-slave-glgz"
server_key: "687ebc91e3b7c3ce1662bfa6f4c13a9d"
server_key_iv: "4edee58051d12c4c83116822e0929cc3"
server_key_version: "encryptionkey@0"
I20260927 04:02:19.342464 3657 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 04:02:19.343153 3671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:19.343312 3657 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:19.343405 3657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/ts-2/wal
uuid: "4f8f299464cb4f8d89a0bbd9a244eb7b"
format_stamp: "Formatted at 2026-09-27 04:02:19 on dist-test-slave-glgz"
server_key: "687ebc91e3b7c3ce1662bfa6f4c13a9d"
server_key_iv: "4edee58051d12c4c83116822e0929cc3"
server_key_version: "encryptionkey@0"
I20260927 04:02:19.343482 3657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:02:19.352746 3657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:02:19.353030 3657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:02:19.353165 3657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:02:19.353376 3657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:02:19.353701 3657 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:02:19.353752 3657 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:19.353792 3657 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:02:19.353818 3657 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:19.359332 3657 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.67:35089
I20260927 04:02:19.359405 3784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.67:35089 every 8 connection(s)
I20260927 04:02:19.359720 3657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/ts-2/data/info.pb
I20260927 04:02:19.360662 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 3657
I20260927 04:02:19.360765 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/ts-2/wal/instance
I20260927 04:02:19.360967 16101 external_mini_cluster.cc:1428] Setting key 425496bbc99de9e43c48958cdeeb10b7
I20260927 04:02:19.364940 3375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:19.360455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.67:55053 (local address 127.15.185.126:34467)
0927 04:02:19.360522 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:19.360525 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:19.360772 (+ 247us) server_negotiation.cc:407] Connection header received
0927 04:02:19.361376 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:19.361377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:19.361389 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:19.361437 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:19.362200 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:19.362688 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:19.363728 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:19.363912 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:19.363961 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:19.364719 (+ 758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:19.364721 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:19.364721 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:02:19.364725 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:19.364739 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:02:19.364743 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:19.364786 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:19.364861 (+ 75us) server_negotiation.cc:299] Negotiation successful
0927 04:02:19.364883 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 04:02:19.365160 3787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:19.360444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:34467 (local address 127.15.185.67:55053)
0927 04:02:19.360618 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:19.360659 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:19.361302 (+ 643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:19.361509 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:19.361525 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:19.361536 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:02:19.362125 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:19.362130 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:19.362776 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:19.362776 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:19.363634 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:19.363637 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:19.363977 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:19.363978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:19.364080 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:19.364581 (+ 501us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 04:02:19.364581 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:02:19.364609 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 04:02:19.364630 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:02:19.364795 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:02:19.364798 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:02:19.364800 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 04:02:19.364851 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 04:02:19.364924 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":81,"threads_started":1}
I20260927 04:02:19.365586 3785 heartbeater.cc:347] Connected to a master server at 127.15.185.126:34467
I20260927 04:02:19.365686 3785 heartbeater.cc:464] Registering TS with master...
I20260927 04:02:19.365833 3785 heartbeater.cc:511] Master 127.15.185.126:34467 requested a full tablet report, sending...
I20260927 04:02:19.366182 3335 ts_manager.cc:194] Registered new tserver with Master: 4f8f299464cb4f8d89a0bbd9a244eb7b (127.15.185.67:35089)
I20260927 04:02:19.366508 3335 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.185.67:55053
I20260927 04:02:19.372848 16101 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 04:02:19.376926 3375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:19.374136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50300 (local address 127.15.185.126:34467)
0927 04:02:19.374188 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:19.374189 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:19.374299 (+ 110us) server_negotiation.cc:407] Connection header received
0927 04:02:19.374356 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:19.374357 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:19.374366 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:19.374405 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:19.374616 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:19.375171 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:19.375872 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:19.376059 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:19.376115 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:19.376737 (+ 622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:19.376738 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:19.376738 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:02:19.376742 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:19.376754 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:02:19.376758 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:19.376809 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:19.376852 (+ 43us) server_negotiation.cc:299] Negotiation successful
0927 04:02:19.376870 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 04:02:19.378573 3335 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50300:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 04:02:19.380146 3335 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 04:02:19.386377 3797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:19.382405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35642 (local address 127.15.185.67:35089)
0927 04:02:19.382592 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:19.382597 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:19.382802 (+ 205us) server_negotiation.cc:407] Connection header received
0927 04:02:19.382915 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:19.382918 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:19.382932 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:19.382979 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:02:19.383169 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:19.384239 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:19.385317 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:19.386021 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:19.386118 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:19.386212 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:19.386246 (+ 34us) server_negotiation.cc:299] Negotiation successful
0927 04:02:19.386286 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":94,"threads_started":1}
I20260927 04:02:19.386397 3794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:19.382146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.67:35089 (local address 127.0.0.1:35642)
0927 04:02:19.382748 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:19.382757 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:19.382829 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:19.383078 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:19.383078 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:19.383080 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:02:19.383111 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:19.383112 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:19.384318 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:19.384319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:19.385229 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:19.385231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:19.386115 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:19.386116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:19.386173 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:19.386186 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 04:02:19.386235 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 04:02:19.386300 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"mutex_wait_us":2,"spinlock_wait_cycles":384,"thread_start_us":109,"threads_started":1}
I20260927 04:02:19.386502 3793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:19.382121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.65:33505 (local address 127.0.0.1:34848)
0927 04:02:19.382247 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:19.382286 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:19.382366 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:19.382776 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:19.382779 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:19.382781 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:02:19.382851 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:19.382853 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:19.384195 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:19.384196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:19.385423 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:19.385426 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:19.386284 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:19.386285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:19.386334 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:19.386337 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 04:02:19.386392 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 04:02:19.386431 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":15,"thread_start_us":77,"threads_started":1}
I20260927 04:02:19.386765 3796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:19.382416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34848 (local address 127.15.185.65:33505)
0927 04:02:19.382587 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:19.382592 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:19.382603 (+ 11us) server_negotiation.cc:407] Connection header received
0927 04:02:19.382644 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:19.382647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:19.382663 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:19.382928 (+ 265us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:02:19.382989 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:19.384119 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:19.385508 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:19.386207 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:19.386277 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:19.386354 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:19.386641 (+ 287us) server_negotiation.cc:299] Negotiation successful
0927 04:02:19.386684 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":73,"threads_started":1}
I20260927 04:02:19.387264 3719 tablet_service.cc:1467] Processing CreateTablet for tablet d53b3b6d512f452392a78e6353625c92 (DEFAULT_TABLE table=test-table [id=7cdc1c8fd26647e1b64e3aaa1c2aaaea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:02:19.387387 3457 tablet_service.cc:1467] Processing CreateTablet for tablet d53b3b6d512f452392a78e6353625c92 (DEFAULT_TABLE table=test-table [id=7cdc1c8fd26647e1b64e3aaa1c2aaaea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:02:19.387581 3795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:19.382146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.66:44501 (local address 127.0.0.1:55394)
0927 04:02:19.382601 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:19.382611 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:19.382663 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:19.383007 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:19.383008 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:19.383010 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:02:19.383081 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:19.383083 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:19.384056 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:19.384057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:19.385423 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:19.385426 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:19.387345 (+ 1919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:19.387347 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:19.387408 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:19.387412 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 04:02:19.387463 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 04:02:19.387502 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"mutex_wait_us":36,"thread_start_us":109,"threads_started":1}
I20260927 04:02:19.387653 3719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d53b3b6d512f452392a78e6353625c92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:02:19.387796 3798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:19.382612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55394 (local address 127.15.185.66:44501)
0927 04:02:19.382794 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:19.382798 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:19.382807 (+ 9us) server_negotiation.cc:407] Connection header received
0927 04:02:19.382849 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:19.382851 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:19.382867 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:19.382940 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:02:19.383160 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:19.383978 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:19.386403 (+ 2425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:19.387160 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:19.387228 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:19.387310 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:19.387678 (+ 368us) server_negotiation.cc:299] Negotiation successful
0927 04:02:19.387719 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":87,"threads_started":1}
I20260927 04:02:19.387804 3457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d53b3b6d512f452392a78e6353625c92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:02:19.390125 3799 tablet_bootstrap.cc:492] T d53b3b6d512f452392a78e6353625c92 P 4f8f299464cb4f8d89a0bbd9a244eb7b: Bootstrap starting.
I20260927 04:02:19.390545 3800 tablet_bootstrap.cc:492] T d53b3b6d512f452392a78e6353625c92 P 44aeff1d421c48149fae844365c5754e: Bootstrap starting.
I20260927 04:02:19.390748 3588 tablet_service.cc:1467] Processing CreateTablet for tablet d53b3b6d512f452392a78e6353625c92 (DEFAULT_TABLE table=test-table [id=7cdc1c8fd26647e1b64e3aaa1c2aaaea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:02:19.390909 3799 tablet_bootstrap.cc:654] T d53b3b6d512f452392a78e6353625c92 P 4f8f299464cb4f8d89a0bbd9a244eb7b: Neither blocks nor log segments found. Creating new log.
I20260927 04:02:19.391109 3588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d53b3b6d512f452392a78e6353625c92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:02:19.391242 3800 tablet_bootstrap.cc:654] T d53b3b6d512f452392a78e6353625c92 P 44aeff1d421c48149fae844365c5754e: Neither blocks nor log segments found. Creating new log.
I20260927 04:02:19.391252 3799 log.cc:824] T d53b3b6d512f452392a78e6353625c92 P 4f8f299464cb4f8d89a0bbd9a244eb7b: Log is configured to *not* fsync() on all Append() calls
I20260927 04:02:19.391585 3800 log.cc:824] T d53b3b6d512f452392a78e6353625c92 P 44aeff1d421c48149fae844365c5754e: Log is configured to *not* fsync() on all Append() calls
I20260927 04:02:19.392024 3799 tablet_bootstrap.cc:492] T d53b3b6d512f452392a78e6353625c92 P 4f8f299464cb4f8d89a0bbd9a244eb7b: No bootstrap required, opened a new log
I20260927 04:02:19.392103 3799 ts_tablet_manager.cc:1397] T d53b3b6d512f452392a78e6353625c92 P 4f8f299464cb4f8d89a0bbd9a244eb7b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 04:02:19.392287 3800 tablet_bootstrap.cc:492] T d53b3b6d512f452392a78e6353625c92 P 44aeff1d421c48149fae844365c5754e: No bootstrap required, opened a new log
I20260927 04:02:19.392360 3800 ts_tablet_manager.cc:1397] T d53b3b6d512f452392a78e6353625c92 P 44aeff1d421c48149fae844365c5754e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 04:02:19.394364 3800 raft_consensus.cc:348] T d53b3b6d512f452392a78e6353625c92 P 44aeff1d421c48149fae844365c5754e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44aeff1d421c48149fae844365c5754e" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 33505 } } peers { permanent_uuid: "4f8f299464cb4f8d89a0bbd9a244eb7b" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 35089 } } peers { permanent_uuid: "e7fed43d755c4e3697bf0ad721a80103" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 44501 } }
I20260927 04:02:19.394546 3800 raft_consensus.cc:374] T d53b3b6d512f452392a78e6353625c92 P 44aeff1d421c48149fae844365c5754e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:02:19.394613 3800 raft_consensus.cc:729] T d53b3b6d512f452392a78e6353625c92 P 44aeff1d421c48149fae844365c5754e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44aeff1d421c48149fae844365c5754e, State: Initialized, Role: FOLLOWER
I20260927 04:02:19.394773 3803 tablet_bootstrap.cc:492] T d53b3b6d512f452392a78e6353625c92 P e7fed43d755c4e3697bf0ad721a80103: Bootstrap starting.
I20260927 04:02:19.394809 3800 consensus_queue.cc:260] T d53b3b6d512f452392a78e6353625c92 P 44aeff1d421c48149fae844365c5754e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44aeff1d421c48149fae844365c5754e" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 33505 } } peers { permanent_uuid: "4f8f299464cb4f8d89a0bbd9a244eb7b" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 35089 } } peers { permanent_uuid: "e7fed43d755c4e3697bf0ad721a80103" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 44501 } }
I20260927 04:02:19.395154 3800 ts_tablet_manager.cc:1428] T d53b3b6d512f452392a78e6353625c92 P 44aeff1d421c48149fae844365c5754e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 04:02:19.395210 3523 heartbeater.cc:503] Master 127.15.185.126:34467 was elected leader, sending a full tablet report...
I20260927 04:02:19.395473 3803 tablet_bootstrap.cc:654] T d53b3b6d512f452392a78e6353625c92 P e7fed43d755c4e3697bf0ad721a80103: Neither blocks nor log segments found. Creating new log.
I20260927 04:02:19.395430 3799 raft_consensus.cc:348] T d53b3b6d512f452392a78e6353625c92 P 4f8f299464cb4f8d89a0bbd9a244eb7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44aeff1d421c48149fae844365c5754e" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 33505 } } peers { permanent_uuid: "4f8f299464cb4f8d89a0bbd9a244eb7b" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 35089 } } peers { permanent_uuid: "e7fed43d755c4e3697bf0ad721a80103" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 44501 } }
I20260927 04:02:19.395705 3799 raft_consensus.cc:374] T d53b3b6d512f452392a78e6353625c92 P 4f8f299464cb4f8d89a0bbd9a244eb7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:02:19.395764 3799 raft_consensus.cc:729] T d53b3b6d512f452392a78e6353625c92 P 4f8f299464cb4f8d89a0bbd9a244eb7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f8f299464cb4f8d89a0bbd9a244eb7b, State: Initialized, Role: FOLLOWER
I20260927 04:02:19.395853 3803 log.cc:824] T d53b3b6d512f452392a78e6353625c92 P e7fed43d755c4e3697bf0ad721a80103: Log is configured to *not* fsync() on all Append() calls
I20260927 04:02:19.395881 3799 consensus_queue.cc:260] T d53b3b6d512f452392a78e6353625c92 P 4f8f299464cb4f8d89a0bbd9a244eb7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44aeff1d421c48149fae844365c5754e" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 33505 } } peers { permanent_uuid: "4f8f299464cb4f8d89a0bbd9a244eb7b" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 35089 } } peers { permanent_uuid: "e7fed43d755c4e3697bf0ad721a80103" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 44501 } }
I20260927 04:02:19.396155 3799 ts_tablet_manager.cc:1428] T d53b3b6d512f452392a78e6353625c92 P 4f8f299464cb4f8d89a0bbd9a244eb7b: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260927 04:02:19.396211 3785 heartbeater.cc:503] Master 127.15.185.126:34467 was elected leader, sending a full tablet report...
I20260927 04:02:19.396862 3803 tablet_bootstrap.cc:492] T d53b3b6d512f452392a78e6353625c92 P e7fed43d755c4e3697bf0ad721a80103: No bootstrap required, opened a new log
I20260927 04:02:19.396937 3803 ts_tablet_manager.cc:1397] T d53b3b6d512f452392a78e6353625c92 P e7fed43d755c4e3697bf0ad721a80103: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260927 04:02:19.398808 3803 raft_consensus.cc:348] T d53b3b6d512f452392a78e6353625c92 P e7fed43d755c4e3697bf0ad721a80103 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44aeff1d421c48149fae844365c5754e" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 33505 } } peers { permanent_uuid: "4f8f299464cb4f8d89a0bbd9a244eb7b" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 35089 } } peers { permanent_uuid: "e7fed43d755c4e3697bf0ad721a80103" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 44501 } }
I20260927 04:02:19.398936 3803 raft_consensus.cc:374] T d53b3b6d512f452392a78e6353625c92 P e7fed43d755c4e3697bf0ad721a80103 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:02:19.398957 3803 raft_consensus.cc:729] T d53b3b6d512f452392a78e6353625c92 P e7fed43d755c4e3697bf0ad721a80103 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7fed43d755c4e3697bf0ad721a80103, State: Initialized, Role: FOLLOWER
I20260927 04:02:19.399096 3803 consensus_queue.cc:260] T d53b3b6d512f452392a78e6353625c92 P e7fed43d755c4e3697bf0ad721a80103 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44aeff1d421c48149fae844365c5754e" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 33505 } } peers { permanent_uuid: "4f8f299464cb4f8d89a0bbd9a244eb7b" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 35089 } } peers { permanent_uuid: "e7fed43d755c4e3697bf0ad721a80103" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 44501 } }
I20260927 04:02:19.399330 3803 ts_tablet_manager.cc:1428] T d53b3b6d512f452392a78e6353625c92 P e7fed43d755c4e3697bf0ad721a80103: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 04:02:19.399381 3654 heartbeater.cc:503] Master 127.15.185.126:34467 was elected leader, sending a full tablet report...
W20260927 04:02:19.488586 3655 tablet.cc:2369] T d53b3b6d512f452392a78e6353625c92 P e7fed43d755c4e3697bf0ad721a80103: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 04:02:19.536124 3807 raft_consensus.cc:482] T d53b3b6d512f452392a78e6353625c92 P e7fed43d755c4e3697bf0ad721a80103 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:02:19.536247 3807 raft_consensus.cc:504] T d53b3b6d512f452392a78e6353625c92 P e7fed43d755c4e3697bf0ad721a80103 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44aeff1d421c48149fae844365c5754e" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 33505 } } peers { permanent_uuid: "4f8f299464cb4f8d89a0bbd9a244eb7b" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 35089 } } peers { permanent_uuid: "e7fed43d755c4e3697bf0ad721a80103" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 44501 } }
I20260927 04:02:19.536612 3807 leader_election.cc:288] T d53b3b6d512f452392a78e6353625c92 P e7fed43d755c4e3697bf0ad721a80103 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44aeff1d421c48149fae844365c5754e (127.15.185.65:33505), 4f8f299464cb4f8d89a0bbd9a244eb7b (127.15.185.67:35089)
I20260927 04:02:19.539402 3796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:19.536851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:33207 (local address 127.15.185.65:33505)
0927 04:02:19.536920 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:19.536922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:19.536933 (+ 11us) server_negotiation.cc:407] Connection header received
0927 04:02:19.536974 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:19.536975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:19.536995 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:19.537058 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:02:19.537254 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:19.537728 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:19.538680 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:19.539029 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:19.539135 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:19.539144 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:19.539289 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 04:02:19.539323 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 04:02:19.539439 3656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:19.536788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.65:33505 (local address 127.15.185.66:33207)
0927 04:02:19.536846 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:19.536860 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:19.536908 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:19.537061 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:19.537062 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:19.537063 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:02:19.537142 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:19.537143 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:19.537819 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:19.537820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:19.538580 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:19.538582 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:19.539135 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:19.539136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:19.539216 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:19.539219 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 04:02:19.539283 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 04:02:19.539323 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 04:02:19.539441 3797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:19.536923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.185.66:37703 (local address 127.15.185.67:35089)
0927 04:02:19.536970 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:19.536973 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:19.537021 (+ 48us) server_negotiation.cc:407] Connection header received
0927 04:02:19.537091 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:19.537092 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:19.537101 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:19.537182 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:02:19.537348 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:19.537794 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:19.538697 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:19.539072 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:19.539169 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:19.539177 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:19.539337 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 04:02:19.539375 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 04:02:19.539459 3808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:19.536812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.67:35089 (local address 127.15.185.66:37703)
0927 04:02:19.536967 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:19.536978 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:19.537037 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:19.537185 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:19.537185 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:19.537187 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:02:19.537273 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:19.537274 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:19.537876 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:19.537877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:19.538607 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:19.538609 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:19.539170 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:19.539171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:19.539259 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:19.539262 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 04:02:19.539319 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 04:02:19.539356 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":88,"threads_started":1}
I20260927 04:02:19.539783 3739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d53b3b6d512f452392a78e6353625c92" candidate_uuid: "e7fed43d755c4e3697bf0ad721a80103" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f8f299464cb4f8d89a0bbd9a244eb7b" is_pre_election: true
I20260927 04:02:19.539783 3477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d53b3b6d512f452392a78e6353625c92" candidate_uuid: "e7fed43d755c4e3697bf0ad721a80103" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44aeff1d421c48149fae844365c5754e" is_pre_election: true
I20260927 04:02:19.539951 3477 raft_consensus.cc:2454] T d53b3b6d512f452392a78e6353625c92 P 44aeff1d421c48149fae844365c5754e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7fed43d755c4e3697bf0ad721a80103 in term 0.
I20260927 04:02:19.539963 3739 raft_consensus.cc:2454] T d53b3b6d512f452392a78e6353625c92 P 4f8f299464cb4f8d89a0bbd9a244eb7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7fed43d755c4e3697bf0ad721a80103 in term 0.
I20260927 04:02:19.540189 3544 leader_election.cc:302] T d53b3b6d512f452392a78e6353625c92 P e7fed43d755c4e3697bf0ad721a80103 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f8f299464cb4f8d89a0bbd9a244eb7b, e7fed43d755c4e3697bf0ad721a80103; no voters:
I20260927 04:02:19.540307 3807 raft_consensus.cc:2781] T d53b3b6d512f452392a78e6353625c92 P e7fed43d755c4e3697bf0ad721a80103 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 04:02:19.540361 3807 raft_consensus.cc:482] T d53b3b6d512f452392a78e6353625c92 P e7fed43d755c4e3697bf0ad721a80103 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 04:02:19.540385 3807 raft_consensus.cc:3037] T d53b3b6d512f452392a78e6353625c92 P e7fed43d755c4e3697bf0ad721a80103 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:02:19.541043 3807 raft_consensus.cc:504] T d53b3b6d512f452392a78e6353625c92 P e7fed43d755c4e3697bf0ad721a80103 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44aeff1d421c48149fae844365c5754e" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 33505 } } peers { permanent_uuid: "4f8f299464cb4f8d89a0bbd9a244eb7b" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 35089 } } peers { permanent_uuid: "e7fed43d755c4e3697bf0ad721a80103" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 44501 } }
I20260927 04:02:19.541178 3807 leader_election.cc:288] T d53b3b6d512f452392a78e6353625c92 P e7fed43d755c4e3697bf0ad721a80103 [CANDIDATE]: Term 1 election: Requested vote from peers 44aeff1d421c48149fae844365c5754e (127.15.185.65:33505), 4f8f299464cb4f8d89a0bbd9a244eb7b (127.15.185.67:35089)
I20260927 04:02:19.541330 3477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d53b3b6d512f452392a78e6353625c92" candidate_uuid: "e7fed43d755c4e3697bf0ad721a80103" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44aeff1d421c48149fae844365c5754e"
I20260927 04:02:19.541360 3739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d53b3b6d512f452392a78e6353625c92" candidate_uuid: "e7fed43d755c4e3697bf0ad721a80103" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f8f299464cb4f8d89a0bbd9a244eb7b"
I20260927 04:02:19.541420 3739 raft_consensus.cc:3037] T d53b3b6d512f452392a78e6353625c92 P 4f8f299464cb4f8d89a0bbd9a244eb7b [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:02:19.541414 3477 raft_consensus.cc:3037] T d53b3b6d512f452392a78e6353625c92 P 44aeff1d421c48149fae844365c5754e [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:02:19.542284 3739 raft_consensus.cc:2454] T d53b3b6d512f452392a78e6353625c92 P 4f8f299464cb4f8d89a0bbd9a244eb7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7fed43d755c4e3697bf0ad721a80103 in term 1.
I20260927 04:02:19.542284 3477 raft_consensus.cc:2454] T d53b3b6d512f452392a78e6353625c92 P 44aeff1d421c48149fae844365c5754e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7fed43d755c4e3697bf0ad721a80103 in term 1.
I20260927 04:02:19.542461 3544 leader_election.cc:302] T d53b3b6d512f452392a78e6353625c92 P e7fed43d755c4e3697bf0ad721a80103 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f8f299464cb4f8d89a0bbd9a244eb7b, e7fed43d755c4e3697bf0ad721a80103; no voters:
I20260927 04:02:19.542562 3807 raft_consensus.cc:2781] T d53b3b6d512f452392a78e6353625c92 P e7fed43d755c4e3697bf0ad721a80103 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:02:19.542695 3807 raft_consensus.cc:686] T d53b3b6d512f452392a78e6353625c92 P e7fed43d755c4e3697bf0ad721a80103 [term 1 LEADER]: Becoming Leader. State: Replica: e7fed43d755c4e3697bf0ad721a80103, State: Running, Role: LEADER
I20260927 04:02:19.542793 3807 consensus_queue.cc:237] T d53b3b6d512f452392a78e6353625c92 P e7fed43d755c4e3697bf0ad721a80103 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44aeff1d421c48149fae844365c5754e" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 33505 } } peers { permanent_uuid: "4f8f299464cb4f8d89a0bbd9a244eb7b" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 35089 } } peers { permanent_uuid: "e7fed43d755c4e3697bf0ad721a80103" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 44501 } }
I20260927 04:02:19.543618 3334 catalog_manager.cc:5495] T d53b3b6d512f452392a78e6353625c92 P e7fed43d755c4e3697bf0ad721a80103 reported cstate change: term changed from 0 to 1, leader changed from <none> to e7fed43d755c4e3697bf0ad721a80103 (127.15.185.66). New cstate: current_term: 1 leader_uuid: "e7fed43d755c4e3697bf0ad721a80103" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44aeff1d421c48149fae844365c5754e" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 33505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f8f299464cb4f8d89a0bbd9a244eb7b" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 35089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7fed43d755c4e3697bf0ad721a80103" member_type: VOTER last_known_addr { host: "127.15.185.66" port: 44501 } health_report { overall_health: HEALTHY } } }
I20260927 04:02:19.569885 3798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:19.567753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55398 (local address 127.15.185.66:44501)
0927 04:02:19.567825 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:19.567827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:19.567834 (+ 7us) server_negotiation.cc:407] Connection header received
0927 04:02:19.567873 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:19.567874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:19.567892 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:19.567956 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:02:19.568068 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:19.568532 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:19.569243 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:19.569420 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:19.569468 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:19.569587 (+ 119us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:02:19.569686 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:02:19.569761 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:19.569792 (+ 31us) server_negotiation.cc:299] Negotiation successful
0927 04:02:19.569807 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 04:02:19.571817 3477 raft_consensus.cc:1261] T d53b3b6d512f452392a78e6353625c92 P 44aeff1d421c48149fae844365c5754e [term 1 FOLLOWER]: Refusing update from remote peer e7fed43d755c4e3697bf0ad721a80103: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:02:19.571817 3739 raft_consensus.cc:1261] T d53b3b6d512f452392a78e6353625c92 P 4f8f299464cb4f8d89a0bbd9a244eb7b [term 1 FOLLOWER]: Refusing update from remote peer e7fed43d755c4e3697bf0ad721a80103: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:02:19.572214 3809 consensus_queue.cc:1035] T d53b3b6d512f452392a78e6353625c92 P e7fed43d755c4e3697bf0ad721a80103 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44aeff1d421c48149fae844365c5754e" member_type: VOTER last_known_addr { host: "127.15.185.65" port: 33505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:02:19.572362 3807 consensus_queue.cc:1035] T d53b3b6d512f452392a78e6353625c92 P e7fed43d755c4e3697bf0ad721a80103 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f8f299464cb4f8d89a0bbd9a244eb7b" member_type: VOTER last_known_addr { host: "127.15.185.67" port: 35089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:02:19.574987 3814 mvcc.cc:205] Tried to move back new op lower bound from 7333813205283201024 to 7333813205167943680. Current Snapshot: MvccSnapshot[applied={T|T < 7333813205283201024}]
I20260927 04:02:19.575711 3813 mvcc.cc:205] Tried to move back new op lower bound from 7333813205283201024 to 7333813205167943680. Current Snapshot: MvccSnapshot[applied={T|T < 7333813205283201024}]
I20260927 04:02:19.577140 3333 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50300:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:02:19.577247 3333 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50300:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:02:19.578004 3333 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 10f45f8e57e04904aeb2a97aec4cd8f8: Sending DeleteTablet for 3 replicas of tablet d53b3b6d512f452392a78e6353625c92
I20260927 04:02:19.578326 3719 tablet_service.cc:1514] Processing DeleteTablet for tablet d53b3b6d512f452392a78e6353625c92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:02:19 UTC) from {username='slave'} at 127.0.0.1:35642
I20260927 04:02:19.578349 3588 tablet_service.cc:1514] Processing DeleteTablet for tablet d53b3b6d512f452392a78e6353625c92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:02:19 UTC) from {username='slave'} at 127.0.0.1:55394
I20260927 04:02:19.578553 3457 tablet_service.cc:1514] Processing DeleteTablet for tablet d53b3b6d512f452392a78e6353625c92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:02:19 UTC) from {username='slave'} at 127.0.0.1:34848
I20260927 04:02:19.578620 3821 tablet_replica.cc:330] T d53b3b6d512f452392a78e6353625c92 P 4f8f299464cb4f8d89a0bbd9a244eb7b: stopping tablet replica
I20260927 04:02:19.578747 3821 raft_consensus.cc:2229] T d53b3b6d512f452392a78e6353625c92 P 4f8f299464cb4f8d89a0bbd9a244eb7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:02:19.578817 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 3395
I20260927 04:02:19.578850 3821 raft_consensus.cc:2258] T d53b3b6d512f452392a78e6353625c92 P 4f8f299464cb4f8d89a0bbd9a244eb7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:02:19.578981 3822 tablet_replica.cc:330] T d53b3b6d512f452392a78e6353625c92 P e7fed43d755c4e3697bf0ad721a80103: stopping tablet replica
I20260927 04:02:19.579074 3822 raft_consensus.cc:2229] T d53b3b6d512f452392a78e6353625c92 P e7fed43d755c4e3697bf0ad721a80103 [term 1 LEADER]: Raft consensus shutting down.
I20260927 04:02:19.579149 3822 raft_consensus.cc:2258] T d53b3b6d512f452392a78e6353625c92 P e7fed43d755c4e3697bf0ad721a80103 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:02:19.579689 3822 ts_tablet_manager.cc:1905] T d53b3b6d512f452392a78e6353625c92 P e7fed43d755c4e3697bf0ad721a80103: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:02:19.581198 3821 ts_tablet_manager.cc:1905] T d53b3b6d512f452392a78e6353625c92 P 4f8f299464cb4f8d89a0bbd9a244eb7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:02:19.581331 3822 ts_tablet_manager.cc:1918] T d53b3b6d512f452392a78e6353625c92 P e7fed43d755c4e3697bf0ad721a80103: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:02:19.581406 3822 log.cc:1196] T d53b3b6d512f452392a78e6353625c92 P e7fed43d755c4e3697bf0ad721a80103: Deleting WAL directory at /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/ts-1/wal/wals/d53b3b6d512f452392a78e6353625c92
I20260927 04:02:19.581661 3822 ts_tablet_manager.cc:1939] T d53b3b6d512f452392a78e6353625c92 P e7fed43d755c4e3697bf0ad721a80103: Deleting consensus metadata
I20260927 04:02:19.582050 3321 catalog_manager.cc:4841] TS e7fed43d755c4e3697bf0ad721a80103 (127.15.185.66:44501): tablet d53b3b6d512f452392a78e6353625c92 (table test-table [id=7cdc1c8fd26647e1b64e3aaa1c2aaaea]) successfully deleted
I20260927 04:02:19.582553 3821 ts_tablet_manager.cc:1918] T d53b3b6d512f452392a78e6353625c92 P 4f8f299464cb4f8d89a0bbd9a244eb7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:02:19.582636 3821 log.cc:1196] T d53b3b6d512f452392a78e6353625c92 P 4f8f299464cb4f8d89a0bbd9a244eb7b: Deleting WAL directory at /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481498018407-16101-0/minicluster-data/ts-2/wal/wals/d53b3b6d512f452392a78e6353625c92
I20260927 04:02:19.582893 3821 ts_tablet_manager.cc:1939] T d53b3b6d512f452392a78e6353625c92 P 4f8f299464cb4f8d89a0bbd9a244eb7b: Deleting consensus metadata
I20260927 04:02:19.583299 3322 catalog_manager.cc:4841] TS 4f8f299464cb4f8d89a0bbd9a244eb7b (127.15.185.67:35089): tablet d53b3b6d512f452392a78e6353625c92 (table test-table [id=7cdc1c8fd26647e1b64e3aaa1c2aaaea]) successfully deleted
W20260927 04:02:19.584332 3319 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.185.65:33505 (error 108)
I20260927 04:02:19.584870 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 3526
I20260927 04:02:19.584898 3795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:19.584774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.65:33505 (local address 127.0.0.1:34850)
0927 04:02:19.584826 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:19.584865 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.65:33505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 04:02:19.585831 3319 catalog_manager.cc:4584] TS 44aeff1d421c48149fae844365c5754e (127.15.185.65:33505): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d53b3b6d512f452392a78e6353625c92: Network error: Client connection negotiation failed: client connection to 127.15.185.65:33505: connect: Connection refused (error 111)
I20260927 04:02:19.590425 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 3657
I20260927 04:02:19.595258 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 3304
2026-09-27T04:02:19Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (829 ms)
[----------] 6 tests from EncryptionPolicyTest (8534 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481498018407-16101-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:02:19 dist-test-slave-glgz krb5kdc[3827](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:02:19 dist-test-slave-glgz krb5kdc[3827](info): set up 2 sockets
Sep 27 04:02:19 dist-test-slave-glgz krb5kdc[3827](info): commencing operation
krb5kdc: starting...
W20260927 04:02:20.659525 16101 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Sep 27 04:02:20 dist-test-slave-glgz krb5kdc[3827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:02:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:02:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481498018407-16101-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481498018407-16101-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 04:02:20.781170 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481498018407-16101-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481498018407-16101-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481498018407-16101-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36829
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481498018407-16101-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36829
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:02:20.861092 3840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:02:20.861349 3840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:02:20.861388 3840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:02:20.862538 3840 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:02:20.862607 3840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:02:20.862632 3840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:02:20.862661 3840 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:02:20.862699 3840 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:02:20.864161 3840 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481498018407-16101-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481498018407-16101-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36829
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481498018407-16101-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481498018407-16101-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:02:20.864449 3840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:02:20.864674 3840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:02:20.869644 3846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:20.869688 3845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:20.869644 3848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:20.869777 3840 server_base.cc:1034] running on GCE node
I20260927 04:02:20.870102 3840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:02:20.870402 3840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:02:20.871567 3840 hybrid_clock.cc:648] HybridClock initialized: now 1790481740871551 us; error 43 us; skew 500 ppm
Sep 27 04:02:20 dist-test-slave-glgz krb5kdc[3827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481740, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:02:20.900542 3840 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 04:02:20.900988 3840 webserver.cc:460] Webserver started at http://127.0.0.1:33181/ using document root <none> and password file <none>
I20260927 04:02:20.901190 3840 fs_manager.cc:359] Metadata directory not provided
I20260927 04:02:20.901232 3840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:02:20.901324 3840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:02:20.902376 3840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481498018407-16101-0/minicluster-data/master-0/data/instance:
uuid: "334c410eebb94e22bd1398fed49a44a9"
format_stamp: "Formatted at 2026-09-27 04:02:20 on dist-test-slave-glgz"
server_key: "ea942dacc5739c9429e3420ba6ce8be4"
server_key_iv: "8aa3ebe6cb4cf4c6d87c6d1331f5347b"
server_key_version: "encryptionkey@0"
I20260927 04:02:20.902696 3840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481498018407-16101-0/minicluster-data/master-0/wal/instance:
uuid: "334c410eebb94e22bd1398fed49a44a9"
format_stamp: "Formatted at 2026-09-27 04:02:20 on dist-test-slave-glgz"
server_key: "ea942dacc5739c9429e3420ba6ce8be4"
server_key_iv: "8aa3ebe6cb4cf4c6d87c6d1331f5347b"
server_key_version: "encryptionkey@0"
I20260927 04:02:20.904115 3840 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 04:02:20.904845 3855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:20.905018 3840 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 04:02:20.905113 3840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481498018407-16101-0/minicluster-data/master-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481498018407-16101-0/minicluster-data/master-0/wal
uuid: "334c410eebb94e22bd1398fed49a44a9"
format_stamp: "Formatted at 2026-09-27 04:02:20 on dist-test-slave-glgz"
server_key: "ea942dacc5739c9429e3420ba6ce8be4"
server_key_iv: "8aa3ebe6cb4cf4c6d87c6d1331f5347b"
server_key_version: "encryptionkey@0"
I20260927 04:02:20.905165 3840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481498018407-16101-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481498018407-16101-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481498018407-16101-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:02:20.913465 3840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:02:20.915345 3840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:02:20.915454 3840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:02:20.919191 3840 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36829
I20260927 04:02:20.919261 3907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36829 every 8 connection(s)
I20260927 04:02:20.919600 3840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
I20260927 04:02:20.920436 3908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:02:20.922719 3908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 334c410eebb94e22bd1398fed49a44a9: Bootstrap starting.
I20260927 04:02:20.923375 3908 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 334c410eebb94e22bd1398fed49a44a9: Neither blocks nor log segments found. Creating new log.
I20260927 04:02:20.923648 3908 log.cc:824] T 00000000000000000000000000000000 P 334c410eebb94e22bd1398fed49a44a9: Log is configured to *not* fsync() on all Append() calls
I20260927 04:02:20.924310 3908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 334c410eebb94e22bd1398fed49a44a9: No bootstrap required, opened a new log
I20260927 04:02:20.925868 3908 raft_consensus.cc:348] T 00000000000000000000000000000000 P 334c410eebb94e22bd1398fed49a44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "334c410eebb94e22bd1398fed49a44a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36829 } }
I20260927 04:02:20.926026 3908 raft_consensus.cc:374] T 00000000000000000000000000000000 P 334c410eebb94e22bd1398fed49a44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:02:20.926083 3908 raft_consensus.cc:729] T 00000000000000000000000000000000 P 334c410eebb94e22bd1398fed49a44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 334c410eebb94e22bd1398fed49a44a9, State: Initialized, Role: FOLLOWER
I20260927 04:02:20.926223 3908 consensus_queue.cc:260] T 00000000000000000000000000000000 P 334c410eebb94e22bd1398fed49a44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "334c410eebb94e22bd1398fed49a44a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36829 } }
I20260927 04:02:20.926301 3908 raft_consensus.cc:388] T 00000000000000000000000000000000 P 334c410eebb94e22bd1398fed49a44a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:02:20.926343 3908 raft_consensus.cc:482] T 00000000000000000000000000000000 P 334c410eebb94e22bd1398fed49a44a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:02:20.926347 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 3840
I20260927 04:02:20.926391 3908 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 334c410eebb94e22bd1398fed49a44a9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:02:20.926469 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481498018407-16101-0/minicluster-data/master-0/wal/instance
I20260927 04:02:20.926667 16101 external_mini_cluster.cc:1428] Setting key c0be0786ef59b6be03c968218ce4a1ce
I20260927 04:02:20.927016 3908 raft_consensus.cc:504] T 00000000000000000000000000000000 P 334c410eebb94e22bd1398fed49a44a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "334c410eebb94e22bd1398fed49a44a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36829 } }
I20260927 04:02:20.927155 3908 leader_election.cc:302] T 00000000000000000000000000000000 P 334c410eebb94e22bd1398fed49a44a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 334c410eebb94e22bd1398fed49a44a9; no voters:
I20260927 04:02:20.927321 3908 leader_election.cc:288] T 00000000000000000000000000000000 P 334c410eebb94e22bd1398fed49a44a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:02:20.927350 3913 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 334c410eebb94e22bd1398fed49a44a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:02:20.927539 3913 raft_consensus.cc:686] T 00000000000000000000000000000000 P 334c410eebb94e22bd1398fed49a44a9 [term 1 LEADER]: Becoming Leader. State: Replica: 334c410eebb94e22bd1398fed49a44a9, State: Running, Role: LEADER
I20260927 04:02:20.927579 3908 sys_catalog.cc:564] T 00000000000000000000000000000000 P 334c410eebb94e22bd1398fed49a44a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:02:20.927733 3913 consensus_queue.cc:237] T 00000000000000000000000000000000 P 334c410eebb94e22bd1398fed49a44a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "334c410eebb94e22bd1398fed49a44a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36829 } }
I20260927 04:02:20.928287 3915 sys_catalog.cc:455] T 00000000000000000000000000000000 P 334c410eebb94e22bd1398fed49a44a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 334c410eebb94e22bd1398fed49a44a9. Latest consensus state: current_term: 1 leader_uuid: "334c410eebb94e22bd1398fed49a44a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "334c410eebb94e22bd1398fed49a44a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36829 } } }
I20260927 04:02:20.928372 3915 sys_catalog.cc:458] T 00000000000000000000000000000000 P 334c410eebb94e22bd1398fed49a44a9 [sys.catalog]: This master's current role is: LEADER
I20260927 04:02:20.928524 3914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 334c410eebb94e22bd1398fed49a44a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "334c410eebb94e22bd1398fed49a44a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "334c410eebb94e22bd1398fed49a44a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36829 } } }
I20260927 04:02:20.928637 3914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 334c410eebb94e22bd1398fed49a44a9 [sys.catalog]: This master's current role is: LEADER
I20260927 04:02:20.929138 3922 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:02:20.929726 3922 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:02:20.931303 3922 catalog_manager.cc:1347] Generated new cluster ID: 525bb0a4e9094deea3857303c9449414
I20260927 04:02:20.931357 3922 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 04:02:20 dist-test-slave-glgz krb5kdc[3827](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 04:02:20.937834 3912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:20.927162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56959 (local address 127.0.0.1:36829)
0927 04:02:20.927343 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:20.927354 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:20.927532 (+ 178us) server_negotiation.cc:407] Connection header received
0927 04:02:20.927648 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:20.927655 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:20.927709 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:20.927835 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:20.928751 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:20.930168 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:20.931251 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:20.931733 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:20.931849 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:20.934180 (+ 2331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:20.934184 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:20.934185 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:20.934203 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:20.936227 (+ 2024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:20.936637 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:20.936640 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:20.936640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:20.936736 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:20.936957 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:20.936959 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:20.936959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:20.937190 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:20.937405 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:20.937490 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 04:02:20.937596 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":99,"threads_started":1}
I20260927 04:02:20.938869 16101 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 04:02:20.949541 3922 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:02:20.950304 3922 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:02:20.953955 3912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:20.940274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47457 (local address 127.0.0.1:36829)
0927 04:02:20.940312 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:20.940314 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:20.947349 (+ 7035us) server_negotiation.cc:407] Connection header received
0927 04:02:20.947430 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:20.947431 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:20.947445 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:20.947498 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:20.948111 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:20.948686 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:20.949613 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:20.949881 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:20.949968 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:20.951323 (+ 1355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:20.951325 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:20.951326 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:20.951333 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:20.952758 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:20.953135 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:20.953136 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:20.953136 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:20.953195 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:20.953423 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:20.953424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:20.953424 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:20.953540 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:20.953653 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:20.953779 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 04:02:20.953844 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 04:02:20.954732 3922 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 334c410eebb94e22bd1398fed49a44a9: Generated new TSK 0
I20260927 04:02:20.954908 3922 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 04:02:20.957446 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 3840
W20260927 04:02:20.961436 3825 connection.cc:535] client connection to 127.0.0.1:36829 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36829): Connection reset by peer (error 104)
2026-09-27T04:02:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1359 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481498018407-16101-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:02:21 dist-test-slave-glgz krb5kdc[3939](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:02:21 dist-test-slave-glgz krb5kdc[3939](info): set up 2 sockets
Sep 27 04:02:21 dist-test-slave-glgz krb5kdc[3939](info): commencing operation
krb5kdc: starting...
W20260927 04:02:21.995379 16101 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Sep 27 04:02:22 dist-test-slave-glgz krb5kdc[3939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:02:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:02:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481498018407-16101-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481498018407-16101-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 04:02:22.118227 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481498018407-16101-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481498018407-16101-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481498018407-16101-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35283
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481498018407-16101-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35283
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:02:22.201581 3952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:02:22.201953 3952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:02:22.202044 3952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:02:22.203758 3952 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:02:22.203845 3952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:02:22.203874 3952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:02:22.203927 3952 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:02:22.203960 3952 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:02:22.205521 3952 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481498018407-16101-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481498018407-16101-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35283
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481498018407-16101-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481498018407-16101-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:02:22.205799 3952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:02:22.206022 3952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:02:22.211350 3960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:22.211506 3952 server_base.cc:1034] running on GCE node
W20260927 04:02:22.211396 3957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:22.211436 3958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:22.211889 3952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:02:22.212152 3952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:02:22.213291 3952 hybrid_clock.cc:648] HybridClock initialized: now 1790481742213280 us; error 30 us; skew 500 ppm
Sep 27 04:02:22 dist-test-slave-glgz krb5kdc[3939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481742, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:02:22.215735 3952 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 04:02:22.216164 3952 webserver.cc:460] Webserver started at http://127.0.0.1:35727/ using document root <none> and password file <none>
I20260927 04:02:22.216387 3952 fs_manager.cc:359] Metadata directory not provided
I20260927 04:02:22.216429 3952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:02:22.216573 3952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:02:22.217693 3952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481498018407-16101-0/minicluster-data/master-0/data/instance:
uuid: "372fc19b3803450b8d89c54d3e981d2c"
format_stamp: "Formatted at 2026-09-27 04:02:22 on dist-test-slave-glgz"
server_key: "fb6f5c5cb40d07a441c5c081f2d3fd1a"
server_key_iv: "fd48b8f32d3ccef2b925a46a0af0ae62"
server_key_version: "encryptionkey@0"
I20260927 04:02:22.218052 3952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481498018407-16101-0/minicluster-data/master-0/wal/instance:
uuid: "372fc19b3803450b8d89c54d3e981d2c"
format_stamp: "Formatted at 2026-09-27 04:02:22 on dist-test-slave-glgz"
server_key: "fb6f5c5cb40d07a441c5c081f2d3fd1a"
server_key_iv: "fd48b8f32d3ccef2b925a46a0af0ae62"
server_key_version: "encryptionkey@0"
I20260927 04:02:22.219420 3952 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 04:02:22.220191 3967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:22.220381 3952 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 04:02:22.220471 3952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481498018407-16101-0/minicluster-data/master-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481498018407-16101-0/minicluster-data/master-0/wal
uuid: "372fc19b3803450b8d89c54d3e981d2c"
format_stamp: "Formatted at 2026-09-27 04:02:22 on dist-test-slave-glgz"
server_key: "fb6f5c5cb40d07a441c5c081f2d3fd1a"
server_key_iv: "fd48b8f32d3ccef2b925a46a0af0ae62"
server_key_version: "encryptionkey@0"
I20260927 04:02:22.220543 3952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481498018407-16101-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481498018407-16101-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481498018407-16101-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:02:22.243294 3952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:02:22.245622 3952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:02:22.245841 3952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:02:22.249743 3952 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35283
I20260927 04:02:22.250136 3952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
I20260927 04:02:22.250138 4019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35283 every 8 connection(s)
I20260927 04:02:22.250856 4020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:02:22.253232 4020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 372fc19b3803450b8d89c54d3e981d2c: Bootstrap starting.
I20260927 04:02:22.253258 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 3952
I20260927 04:02:22.253384 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481498018407-16101-0/minicluster-data/master-0/wal/instance
I20260927 04:02:22.253588 16101 external_mini_cluster.cc:1428] Setting key d14576769e272d8e6befeaabd8f9d730
I20260927 04:02:22.253944 4020 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 372fc19b3803450b8d89c54d3e981d2c: Neither blocks nor log segments found. Creating new log.
I20260927 04:02:22.254314 4020 log.cc:824] T 00000000000000000000000000000000 P 372fc19b3803450b8d89c54d3e981d2c: Log is configured to *not* fsync() on all Append() calls
I20260927 04:02:22.255190 4020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 372fc19b3803450b8d89c54d3e981d2c: No bootstrap required, opened a new log
I20260927 04:02:22.257602 4020 raft_consensus.cc:348] T 00000000000000000000000000000000 P 372fc19b3803450b8d89c54d3e981d2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "372fc19b3803450b8d89c54d3e981d2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35283 } }
I20260927 04:02:22.257751 4020 raft_consensus.cc:374] T 00000000000000000000000000000000 P 372fc19b3803450b8d89c54d3e981d2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:02:22.257776 4020 raft_consensus.cc:729] T 00000000000000000000000000000000 P 372fc19b3803450b8d89c54d3e981d2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 372fc19b3803450b8d89c54d3e981d2c, State: Initialized, Role: FOLLOWER
I20260927 04:02:22.257908 4020 consensus_queue.cc:260] T 00000000000000000000000000000000 P 372fc19b3803450b8d89c54d3e981d2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "372fc19b3803450b8d89c54d3e981d2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35283 } }
I20260927 04:02:22.258001 4020 raft_consensus.cc:388] T 00000000000000000000000000000000 P 372fc19b3803450b8d89c54d3e981d2c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:02:22.258034 4020 raft_consensus.cc:482] T 00000000000000000000000000000000 P 372fc19b3803450b8d89c54d3e981d2c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:02:22.258080 4020 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 372fc19b3803450b8d89c54d3e981d2c [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:02:22.258773 4020 raft_consensus.cc:504] T 00000000000000000000000000000000 P 372fc19b3803450b8d89c54d3e981d2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "372fc19b3803450b8d89c54d3e981d2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35283 } }
I20260927 04:02:22.258891 4020 leader_election.cc:302] T 00000000000000000000000000000000 P 372fc19b3803450b8d89c54d3e981d2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 372fc19b3803450b8d89c54d3e981d2c; no voters:
I20260927 04:02:22.259068 4020 leader_election.cc:288] T 00000000000000000000000000000000 P 372fc19b3803450b8d89c54d3e981d2c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:02:22.259109 4025 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 372fc19b3803450b8d89c54d3e981d2c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:02:22.259279 4025 raft_consensus.cc:686] T 00000000000000000000000000000000 P 372fc19b3803450b8d89c54d3e981d2c [term 1 LEADER]: Becoming Leader. State: Replica: 372fc19b3803450b8d89c54d3e981d2c, State: Running, Role: LEADER
I20260927 04:02:22.259317 4020 sys_catalog.cc:564] T 00000000000000000000000000000000 P 372fc19b3803450b8d89c54d3e981d2c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:02:22.259392 4025 consensus_queue.cc:237] T 00000000000000000000000000000000 P 372fc19b3803450b8d89c54d3e981d2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "372fc19b3803450b8d89c54d3e981d2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35283 } }
I20260927 04:02:22.259747 4026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 372fc19b3803450b8d89c54d3e981d2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "372fc19b3803450b8d89c54d3e981d2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "372fc19b3803450b8d89c54d3e981d2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35283 } } }
I20260927 04:02:22.259891 4027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 372fc19b3803450b8d89c54d3e981d2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 372fc19b3803450b8d89c54d3e981d2c. Latest consensus state: current_term: 1 leader_uuid: "372fc19b3803450b8d89c54d3e981d2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "372fc19b3803450b8d89c54d3e981d2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35283 } } }
I20260927 04:02:22.259965 4026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 372fc19b3803450b8d89c54d3e981d2c [sys.catalog]: This master's current role is: LEADER
I20260927 04:02:22.260057 4027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 372fc19b3803450b8d89c54d3e981d2c [sys.catalog]: This master's current role is: LEADER
I20260927 04:02:22.260581 4037 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:02:22.261068 4037 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:02:22.261288 4023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:22.254181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58637 (local address 127.0.0.1:35283)
0927 04:02:22.254380 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:22.254390 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:22.254422 (+ 32us) server_negotiation.cc:407] Connection header received
0927 04:02:22.255231 (+ 809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:22.255241 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:22.255309 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:22.255462 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:22.255647 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:22.257131 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:22.257986 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:22.258620 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:22.258736 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:22.260407 (+ 1671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:22.260411 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:22.260412 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:02:22.260656 (+ 244us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:22.260697 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:02:22.260713 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:22.260801 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:22.260951 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 04:02:22.260980 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":78,"threads_started":1}
I20260927 04:02:22.262410 16101 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 04:02:22.263458 4037 catalog_manager.cc:1347] Generated new cluster ID: 685a8d1e596a44618f7d0a44795f4a14
I20260927 04:02:22.263515 4037 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:02:22.274217 4023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:22.263817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34517 (local address 127.0.0.1:35283)
0927 04:02:22.263855 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:22.263858 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:22.271280 (+ 7422us) server_negotiation.cc:407] Connection header received
0927 04:02:22.271368 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:22.271369 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:22.271382 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:22.271452 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:22.271610 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:22.272159 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:22.272923 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:22.273138 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:22.273217 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:22.273937 (+ 720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:22.273938 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:22.273939 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:02:22.273944 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:22.273960 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:02:22.273965 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:22.274036 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:22.274102 (+ 66us) server_negotiation.cc:299] Negotiation successful
0927 04:02:22.274112 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 04:02:22.275295 4037 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:02:22.275822 4037 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:02:22.280862 4037 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 372fc19b3803450b8d89c54d3e981d2c: Generated new TSK 0
I20260927 04:02:22.281071 4037 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 04:02:22.287776 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 3952
W20260927 04:02:22.291879 3937 connection.cc:535] client connection to 127.0.0.1:35283 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35283): Connection reset by peer (error 104)
2026-09-27T04:02:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1330 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481498018407-16101-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:02:22 dist-test-slave-glgz krb5kdc[4051](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:02:22 dist-test-slave-glgz krb5kdc[4051](info): set up 2 sockets
Sep 27 04:02:22 dist-test-slave-glgz krb5kdc[4051](info): commencing operation
krb5kdc: starting...
W20260927 04:02:23.323622 16101 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Sep 27 04:02:23 dist-test-slave-glgz krb5kdc[4051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:02:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:02:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481498018407-16101-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481498018407-16101-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 04:02:23.441973 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481498018407-16101-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481498018407-16101-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481498018407-16101-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42733
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481498018407-16101-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42733
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:02:23.524030 4064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:02:23.524296 4064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:02:23.524338 4064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:02:23.525532 4064 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:02:23.525615 4064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:02:23.525640 4064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:02:23.525661 4064 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:02:23.525692 4064 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:02:23.527254 4064 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481498018407-16101-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481498018407-16101-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42733
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481498018407-16101-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481498018407-16101-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:02:23.527565 4064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:02:23.527796 4064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:02:23.532683 4069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:23.532714 4070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:23.532907 4064 server_base.cc:1034] running on GCE node
W20260927 04:02:23.532964 4072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:23.533214 4064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:02:23.533499 4064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:02:23.534682 4064 hybrid_clock.cc:648] HybridClock initialized: now 1790481743534673 us; error 38 us; skew 500 ppm
Sep 27 04:02:23 dist-test-slave-glgz krb5kdc[4051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481743, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:02:23.537395 4064 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 04:02:23.537789 4064 webserver.cc:460] Webserver started at http://127.0.0.1:37087/ using document root <none> and password file <none>
I20260927 04:02:23.537993 4064 fs_manager.cc:359] Metadata directory not provided
I20260927 04:02:23.538036 4064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:02:23.538130 4064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:02:23.539217 4064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481498018407-16101-0/minicluster-data/master-0/data/instance:
uuid: "a4180e2261ab4f998cc97036ea476b6a"
format_stamp: "Formatted at 2026-09-27 04:02:23 on dist-test-slave-glgz"
server_key: "10ce8c24eb53a17b7d2505549d0c69d1"
server_key_iv: "6b77ee77977ec304b28f0eae3a3e095d"
server_key_version: "encryptionkey@0"
I20260927 04:02:23.539517 4064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481498018407-16101-0/minicluster-data/master-0/wal/instance:
uuid: "a4180e2261ab4f998cc97036ea476b6a"
format_stamp: "Formatted at 2026-09-27 04:02:23 on dist-test-slave-glgz"
server_key: "10ce8c24eb53a17b7d2505549d0c69d1"
server_key_iv: "6b77ee77977ec304b28f0eae3a3e095d"
server_key_version: "encryptionkey@0"
I20260927 04:02:23.540789 4064 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:02:23.541509 4079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:23.541728 4064 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:02:23.541814 4064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481498018407-16101-0/minicluster-data/master-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481498018407-16101-0/minicluster-data/master-0/wal
uuid: "a4180e2261ab4f998cc97036ea476b6a"
format_stamp: "Formatted at 2026-09-27 04:02:23 on dist-test-slave-glgz"
server_key: "10ce8c24eb53a17b7d2505549d0c69d1"
server_key_iv: "6b77ee77977ec304b28f0eae3a3e095d"
server_key_version: "encryptionkey@0"
I20260927 04:02:23.541869 4064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481498018407-16101-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481498018407-16101-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481498018407-16101-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:02:23.545737 4064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:02:23.548053 4064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:02:23.548207 4064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:02:23.551879 4064 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42733
I20260927 04:02:23.551944 4131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42733 every 8 connection(s)
I20260927 04:02:23.552260 4064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
I20260927 04:02:23.553131 4132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:02:23.555408 4132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4180e2261ab4f998cc97036ea476b6a: Bootstrap starting.
I20260927 04:02:23.555923 4132 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4180e2261ab4f998cc97036ea476b6a: Neither blocks nor log segments found. Creating new log.
I20260927 04:02:23.556147 4132 log.cc:824] T 00000000000000000000000000000000 P a4180e2261ab4f998cc97036ea476b6a: Log is configured to *not* fsync() on all Append() calls
I20260927 04:02:23.556565 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 4064
I20260927 04:02:23.556703 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481498018407-16101-0/minicluster-data/master-0/wal/instance
I20260927 04:02:23.556850 4132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4180e2261ab4f998cc97036ea476b6a: No bootstrap required, opened a new log
I20260927 04:02:23.556914 16101 external_mini_cluster.cc:1428] Setting key 3ae4a60ec1798b51570f2f7eb72643fb
I20260927 04:02:23.558545 4132 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4180e2261ab4f998cc97036ea476b6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4180e2261ab4f998cc97036ea476b6a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42733 } }
I20260927 04:02:23.558719 4132 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4180e2261ab4f998cc97036ea476b6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:02:23.558809 4132 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4180e2261ab4f998cc97036ea476b6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4180e2261ab4f998cc97036ea476b6a, State: Initialized, Role: FOLLOWER
I20260927 04:02:23.558950 4132 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4180e2261ab4f998cc97036ea476b6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4180e2261ab4f998cc97036ea476b6a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42733 } }
I20260927 04:02:23.559074 4132 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4180e2261ab4f998cc97036ea476b6a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:02:23.559124 4132 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4180e2261ab4f998cc97036ea476b6a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:02:23.559186 4132 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4180e2261ab4f998cc97036ea476b6a [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:02:23.559844 4136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:23.557396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53245 (local address 127.0.0.1:42733)
0927 04:02:23.557607 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:23.557614 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:23.557637 (+ 23us) server_negotiation.cc:407] Connection header received
0927 04:02:23.558524 (+ 887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:23.558531 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:23.558556 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:23.558663 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:23.559388 (+ 725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:23.559391 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:23.559392 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:02:23.559395 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:23.559430 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:02:23.559441 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:23.559494 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:23.559589 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 04:02:23.559620 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":91,"threads_started":1}
I20260927 04:02:23.560300 4132 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4180e2261ab4f998cc97036ea476b6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4180e2261ab4f998cc97036ea476b6a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42733 } }
I20260927 04:02:23.560420 4132 leader_election.cc:302] T 00000000000000000000000000000000 P a4180e2261ab4f998cc97036ea476b6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4180e2261ab4f998cc97036ea476b6a; no voters:
I20260927 04:02:23.560621 4132 leader_election.cc:288] T 00000000000000000000000000000000 P a4180e2261ab4f998cc97036ea476b6a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:02:23.560659 4137 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4180e2261ab4f998cc97036ea476b6a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:02:23.560822 4137 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4180e2261ab4f998cc97036ea476b6a [term 1 LEADER]: Becoming Leader. State: Replica: a4180e2261ab4f998cc97036ea476b6a, State: Running, Role: LEADER
I20260927 04:02:23.560824 16101 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 04:02:23.560854 4132 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4180e2261ab4f998cc97036ea476b6a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:02:23.560910 4137 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4180e2261ab4f998cc97036ea476b6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4180e2261ab4f998cc97036ea476b6a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42733 } }
I20260927 04:02:23.561201 4137 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4180e2261ab4f998cc97036ea476b6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4180e2261ab4f998cc97036ea476b6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4180e2261ab4f998cc97036ea476b6a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42733 } } }
I20260927 04:02:23.561287 4137 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4180e2261ab4f998cc97036ea476b6a [sys.catalog]: This master's current role is: LEADER
I20260927 04:02:23.562382 4142 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:02:23.561537 4139 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4180e2261ab4f998cc97036ea476b6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4180e2261ab4f998cc97036ea476b6a. Latest consensus state: current_term: 1 leader_uuid: "a4180e2261ab4f998cc97036ea476b6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4180e2261ab4f998cc97036ea476b6a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42733 } } }
I20260927 04:02:23.562616 4139 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4180e2261ab4f998cc97036ea476b6a [sys.catalog]: This master's current role is: LEADER
I20260927 04:02:23.562920 4142 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:02:23.563766 4136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:23.562292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51599 (local address 127.0.0.1:42733)
0927 04:02:23.562326 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:23.562328 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:23.562811 (+ 483us) server_negotiation.cc:407] Connection header received
0927 04:02:23.562881 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:23.562882 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:23.562888 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:23.562937 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:23.563544 (+ 607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:23.563546 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:23.563546 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:02:23.563550 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:23.563563 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:02:23.563567 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:23.563624 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:23.563689 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 04:02:23.563702 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 04:02:23.564587 4142 catalog_manager.cc:1347] Generated new cluster ID: 458d5a296b224885b2d3e14dc12caf24
I20260927 04:02:23.564637 4142 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:02:23.596576 4142 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:02:23.596818 4142 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:02:23.600708 4142 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4180e2261ab4f998cc97036ea476b6a: Generated new TSK 0
I20260927 04:02:23.600883 4142 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 04:02:23.608355 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 4064
2026-09-27T04:02:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1320 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481498018407-16101-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:02:23 dist-test-slave-glgz krb5kdc[4163](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:02:23 dist-test-slave-glgz krb5kdc[4163](info): set up 2 sockets
Sep 27 04:02:23 dist-test-slave-glgz krb5kdc[4163](info): commencing operation
krb5kdc: starting...
W20260927 04:02:24.660002 16101 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Sep 27 04:02:24 dist-test-slave-glgz krb5kdc[4163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:02:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:02:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481498018407-16101-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481498018407-16101-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 04:02:24.781913 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481498018407-16101-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481498018407-16101-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481498018407-16101-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32959
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481498018407-16101-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32959
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:02:24.862901 4176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:02:24.863196 4176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:02:24.863226 4176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:02:24.864406 4176 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:02:24.864450 4176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:02:24.864465 4176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:02:24.864478 4176 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:02:24.864492 4176 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:02:24.865953 4176 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481498018407-16101-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481498018407-16101-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481498018407-16101-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481498018407-16101-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:02:24.866217 4176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:02:24.866400 4176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:02:24.871378 4182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:24.871361 4181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:24.871356 4184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:24.871538 4176 server_base.cc:1034] running on GCE node
I20260927 04:02:24.871868 4176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:02:24.872169 4176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:02:24.873332 4176 hybrid_clock.cc:648] HybridClock initialized: now 1790481744873319 us; error 29 us; skew 500 ppm
Sep 27 04:02:24 dist-test-slave-glgz krb5kdc[4163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481744, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:02:24.875936 4176 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 04:02:24.876446 4176 webserver.cc:460] Webserver started at http://127.0.0.1:37487/ using document root <none> and password file <none>
I20260927 04:02:24.876703 4176 fs_manager.cc:359] Metadata directory not provided
I20260927 04:02:24.876782 4176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:02:24.876926 4176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:02:24.878031 4176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481498018407-16101-0/minicluster-data/master-0/data/instance:
uuid: "d2a84150e9624a6a97f83c7c3c0ee3ed"
format_stamp: "Formatted at 2026-09-27 04:02:24 on dist-test-slave-glgz"
server_key: "65ef84ade9d9d71e7e334cf2adb3e587"
server_key_iv: "2deb541a6136ce17d0351321ea2f6cc1"
server_key_version: "encryptionkey@0"
I20260927 04:02:24.878414 4176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481498018407-16101-0/minicluster-data/master-0/wal/instance:
uuid: "d2a84150e9624a6a97f83c7c3c0ee3ed"
format_stamp: "Formatted at 2026-09-27 04:02:24 on dist-test-slave-glgz"
server_key: "65ef84ade9d9d71e7e334cf2adb3e587"
server_key_iv: "2deb541a6136ce17d0351321ea2f6cc1"
server_key_version: "encryptionkey@0"
I20260927 04:02:24.879828 4176 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 04:02:24.880579 4191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:24.880846 4176 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:02:24.880964 4176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481498018407-16101-0/minicluster-data/master-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481498018407-16101-0/minicluster-data/master-0/wal
uuid: "d2a84150e9624a6a97f83c7c3c0ee3ed"
format_stamp: "Formatted at 2026-09-27 04:02:24 on dist-test-slave-glgz"
server_key: "65ef84ade9d9d71e7e334cf2adb3e587"
server_key_iv: "2deb541a6136ce17d0351321ea2f6cc1"
server_key_version: "encryptionkey@0"
I20260927 04:02:24.881067 4176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481498018407-16101-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481498018407-16101-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481498018407-16101-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:02:24.890178 4176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:02:24.892128 4176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:02:24.892289 4176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:02:24.896374 4176 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32959
I20260927 04:02:24.896409 4243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32959 every 8 connection(s)
I20260927 04:02:24.896780 4176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
I20260927 04:02:24.897603 4244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:02:24.899881 4244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2a84150e9624a6a97f83c7c3c0ee3ed: Bootstrap starting.
I20260927 04:02:24.900465 4244 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2a84150e9624a6a97f83c7c3c0ee3ed: Neither blocks nor log segments found. Creating new log.
I20260927 04:02:24.900756 4244 log.cc:824] T 00000000000000000000000000000000 P d2a84150e9624a6a97f83c7c3c0ee3ed: Log is configured to *not* fsync() on all Append() calls
I20260927 04:02:24.901455 4244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2a84150e9624a6a97f83c7c3c0ee3ed: No bootstrap required, opened a new log
I20260927 04:02:24.903023 4244 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2a84150e9624a6a97f83c7c3c0ee3ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2a84150e9624a6a97f83c7c3c0ee3ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32959 } }
I20260927 04:02:24.903260 4244 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2a84150e9624a6a97f83c7c3c0ee3ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:02:24.903311 4244 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2a84150e9624a6a97f83c7c3c0ee3ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2a84150e9624a6a97f83c7c3c0ee3ed, State: Initialized, Role: FOLLOWER
I20260927 04:02:24.903475 4244 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2a84150e9624a6a97f83c7c3c0ee3ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2a84150e9624a6a97f83c7c3c0ee3ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32959 } }
I20260927 04:02:24.903553 4244 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2a84150e9624a6a97f83c7c3c0ee3ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:02:24.903592 4244 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2a84150e9624a6a97f83c7c3c0ee3ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:02:24.903644 4244 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2a84150e9624a6a97f83c7c3c0ee3ed [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:02:24.904151 4244 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2a84150e9624a6a97f83c7c3c0ee3ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2a84150e9624a6a97f83c7c3c0ee3ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32959 } }
I20260927 04:02:24.904270 4244 leader_election.cc:302] T 00000000000000000000000000000000 P d2a84150e9624a6a97f83c7c3c0ee3ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2a84150e9624a6a97f83c7c3c0ee3ed; no voters:
I20260927 04:02:24.904452 4244 leader_election.cc:288] T 00000000000000000000000000000000 P d2a84150e9624a6a97f83c7c3c0ee3ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:02:24.904486 4247 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2a84150e9624a6a97f83c7c3c0ee3ed [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:02:24.904670 4247 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2a84150e9624a6a97f83c7c3c0ee3ed [term 1 LEADER]: Becoming Leader. State: Replica: d2a84150e9624a6a97f83c7c3c0ee3ed, State: Running, Role: LEADER
I20260927 04:02:24.904712 4244 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2a84150e9624a6a97f83c7c3c0ee3ed [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:02:24.904806 4247 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2a84150e9624a6a97f83c7c3c0ee3ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2a84150e9624a6a97f83c7c3c0ee3ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32959 } }
I20260927 04:02:24.905197 4249 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2a84150e9624a6a97f83c7c3c0ee3ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2a84150e9624a6a97f83c7c3c0ee3ed. Latest consensus state: current_term: 1 leader_uuid: "d2a84150e9624a6a97f83c7c3c0ee3ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2a84150e9624a6a97f83c7c3c0ee3ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32959 } } }
I20260927 04:02:24.905340 4248 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2a84150e9624a6a97f83c7c3c0ee3ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2a84150e9624a6a97f83c7c3c0ee3ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2a84150e9624a6a97f83c7c3c0ee3ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32959 } } }
I20260927 04:02:24.905347 4249 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2a84150e9624a6a97f83c7c3c0ee3ed [sys.catalog]: This master's current role is: LEADER
I20260927 04:02:24.905441 4248 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2a84150e9624a6a97f83c7c3c0ee3ed [sys.catalog]: This master's current role is: LEADER
I20260927 04:02:24.905817 4256 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:02:24.906329 4256 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:02:24.906767 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 4176
I20260927 04:02:24.906881 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481498018407-16101-0/minicluster-data/master-0/wal/instance
I20260927 04:02:24.907070 16101 external_mini_cluster.cc:1428] Setting key 4fc5ae87c3f3fd34541966d88799cfad
I20260927 04:02:24.908179 4256 catalog_manager.cc:1347] Generated new cluster ID: 031817dbf7964b1e9fbc1f83509c0b0e
I20260927 04:02:24.908229 4256 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:02:24.913508 4256 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:02:24.914005 4256 catalog_manager.cc:1504] Loading token signing keys...
Sep 27 04:02:24 dist-test-slave-glgz krb5kdc[4163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 04:02:24.918795 4266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:24.907610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52037 (local address 127.0.0.1:32959)
0927 04:02:24.907889 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:24.907898 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:24.907930 (+ 32us) server_negotiation.cc:407] Connection header received
0927 04:02:24.908053 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:24.908060 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:24.908114 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:24.908258 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:24.909138 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:24.910534 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:24.911637 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:24.912241 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:24.912356 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:24.915276 (+ 2920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:24.915280 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:24.915280 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:24.915292 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:24.917211 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:24.917604 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:24.917608 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:24.917608 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:24.917711 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:24.917906 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:24.917907 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:24.917907 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:24.918163 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:24.918305 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:24.918414 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 04:02:24.918565 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":71,"threads_started":1}
I20260927 04:02:24.919975 16101 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 04:02:24.923934 4256 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2a84150e9624a6a97f83c7c3c0ee3ed: Generated new TSK 0
I20260927 04:02:24.924124 4256 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 04:02:24.928866 4266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:24.921420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42049 (local address 127.0.0.1:32959)
0927 04:02:24.921452 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:24.921454 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:24.921586 (+ 132us) server_negotiation.cc:407] Connection header received
0927 04:02:24.921646 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:24.921647 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:24.921655 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:24.921706 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:24.922331 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:24.923123 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:24.924227 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:24.924559 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:24.924623 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:24.925860 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:24.925861 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:24.925861 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:24.925868 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:24.927416 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:24.927870 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:24.927871 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:24.927872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:24.927949 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:24.928231 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:24.928232 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:24.928232 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:24.928368 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:24.928467 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:24.928675 (+ 208us) server_negotiation.cc:299] Negotiation successful
0927 04:02:24.928769 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 04:02:24.931015 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 4176
2026-09-27T04:02:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1323 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481498018407-16101-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:02:24 dist-test-slave-glgz krb5kdc[4275](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:02:24 dist-test-slave-glgz krb5kdc[4275](info): set up 2 sockets
Sep 27 04:02:24 dist-test-slave-glgz krb5kdc[4275](info): commencing operation
krb5kdc: starting...
W20260927 04:02:25.975688 16101 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Sep 27 04:02:26 dist-test-slave-glgz krb5kdc[4275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:02:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:02:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481498018407-16101-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481498018407-16101-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 04:02:26.096114 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481498018407-16101-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481498018407-16101-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481498018407-16101-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38973
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481498018407-16101-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38973
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:02:26.179893 4288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:02:26.180153 4288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:02:26.180193 4288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:02:26.181387 4288 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:02:26.181459 4288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:02:26.181488 4288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:02:26.181524 4288 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:02:26.181552 4288 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:02:26.183140 4288 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481498018407-16101-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481498018407-16101-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38973
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481498018407-16101-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481498018407-16101-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:02:26.183432 4288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:02:26.183676 4288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:02:26.188587 4293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:26.188637 4294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:26.188683 4296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:26.188824 4288 server_base.cc:1034] running on GCE node
I20260927 04:02:26.189088 4288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:02:26.189373 4288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:02:26.190532 4288 hybrid_clock.cc:648] HybridClock initialized: now 1790481746190499 us; error 68 us; skew 500 ppm
Sep 27 04:02:26 dist-test-slave-glgz krb5kdc[4275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481746, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:02:26.198261 4288 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 04:02:26.198710 4288 webserver.cc:460] Webserver started at http://127.0.0.1:46507/ using document root <none> and password file <none>
I20260927 04:02:26.198902 4288 fs_manager.cc:359] Metadata directory not provided
I20260927 04:02:26.198942 4288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:02:26.199031 4288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:02:26.200223 4288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481498018407-16101-0/minicluster-data/master-0/data/instance:
uuid: "65fd18b3299744d8b7a7d3e7e455f17b"
format_stamp: "Formatted at 2026-09-27 04:02:26 on dist-test-slave-glgz"
server_key: "0e27bede96ce101994f0ed1e1c95744b"
server_key_iv: "61e68429de125c19ebeb4d09c09e1f7a"
server_key_version: "encryptionkey@0"
I20260927 04:02:26.200567 4288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481498018407-16101-0/minicluster-data/master-0/wal/instance:
uuid: "65fd18b3299744d8b7a7d3e7e455f17b"
format_stamp: "Formatted at 2026-09-27 04:02:26 on dist-test-slave-glgz"
server_key: "0e27bede96ce101994f0ed1e1c95744b"
server_key_iv: "61e68429de125c19ebeb4d09c09e1f7a"
server_key_version: "encryptionkey@0"
I20260927 04:02:26.201910 4288 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 04:02:26.202663 4303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:26.202859 4288 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:02:26.202936 4288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481498018407-16101-0/minicluster-data/master-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481498018407-16101-0/minicluster-data/master-0/wal
uuid: "65fd18b3299744d8b7a7d3e7e455f17b"
format_stamp: "Formatted at 2026-09-27 04:02:26 on dist-test-slave-glgz"
server_key: "0e27bede96ce101994f0ed1e1c95744b"
server_key_iv: "61e68429de125c19ebeb4d09c09e1f7a"
server_key_version: "encryptionkey@0"
I20260927 04:02:26.202991 4288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481498018407-16101-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481498018407-16101-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481498018407-16101-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:02:26.218484 4288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:02:26.220573 4288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:02:26.220685 4288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:02:26.224751 4288 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38973
I20260927 04:02:26.224833 4355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38973 every 8 connection(s)
I20260927 04:02:26.225157 4288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
I20260927 04:02:26.226116 4356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:02:26.228693 4356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65fd18b3299744d8b7a7d3e7e455f17b: Bootstrap starting.
I20260927 04:02:26.229262 4356 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65fd18b3299744d8b7a7d3e7e455f17b: Neither blocks nor log segments found. Creating new log.
I20260927 04:02:26.229517 4356 log.cc:824] T 00000000000000000000000000000000 P 65fd18b3299744d8b7a7d3e7e455f17b: Log is configured to *not* fsync() on all Append() calls
I20260927 04:02:26.230211 4356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65fd18b3299744d8b7a7d3e7e455f17b: No bootstrap required, opened a new log
I20260927 04:02:26.231120 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 4288
I20260927 04:02:26.231252 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481498018407-16101-0/minicluster-data/master-0/wal/instance
I20260927 04:02:26.231441 16101 external_mini_cluster.cc:1428] Setting key 240d94f4bce43a33bedac73436bf5e61
I20260927 04:02:26.232651 4356 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65fd18b3299744d8b7a7d3e7e455f17b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65fd18b3299744d8b7a7d3e7e455f17b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38973 } }
I20260927 04:02:26.232843 4356 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65fd18b3299744d8b7a7d3e7e455f17b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:02:26.232930 4356 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65fd18b3299744d8b7a7d3e7e455f17b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65fd18b3299744d8b7a7d3e7e455f17b, State: Initialized, Role: FOLLOWER
I20260927 04:02:26.233081 4356 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65fd18b3299744d8b7a7d3e7e455f17b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65fd18b3299744d8b7a7d3e7e455f17b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38973 } }
I20260927 04:02:26.233179 4356 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65fd18b3299744d8b7a7d3e7e455f17b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:02:26.233229 4356 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65fd18b3299744d8b7a7d3e7e455f17b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:02:26.233289 4356 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65fd18b3299744d8b7a7d3e7e455f17b [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:02:26.233911 4356 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65fd18b3299744d8b7a7d3e7e455f17b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65fd18b3299744d8b7a7d3e7e455f17b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38973 } }
I20260927 04:02:26.234041 4356 leader_election.cc:302] T 00000000000000000000000000000000 P 65fd18b3299744d8b7a7d3e7e455f17b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65fd18b3299744d8b7a7d3e7e455f17b; no voters:
I20260927 04:02:26.234249 4356 leader_election.cc:288] T 00000000000000000000000000000000 P 65fd18b3299744d8b7a7d3e7e455f17b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:02:26.234278 4361 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65fd18b3299744d8b7a7d3e7e455f17b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:02:26.234395 4361 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65fd18b3299744d8b7a7d3e7e455f17b [term 1 LEADER]: Becoming Leader. State: Replica: 65fd18b3299744d8b7a7d3e7e455f17b, State: Running, Role: LEADER
I20260927 04:02:26.234535 4356 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65fd18b3299744d8b7a7d3e7e455f17b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:02:26.234543 4361 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65fd18b3299744d8b7a7d3e7e455f17b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65fd18b3299744d8b7a7d3e7e455f17b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38973 } }
I20260927 04:02:26.234942 4361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65fd18b3299744d8b7a7d3e7e455f17b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65fd18b3299744d8b7a7d3e7e455f17b. Latest consensus state: current_term: 1 leader_uuid: "65fd18b3299744d8b7a7d3e7e455f17b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65fd18b3299744d8b7a7d3e7e455f17b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38973 } } }
I20260927 04:02:26.234952 4362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65fd18b3299744d8b7a7d3e7e455f17b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65fd18b3299744d8b7a7d3e7e455f17b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65fd18b3299744d8b7a7d3e7e455f17b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38973 } } }
I20260927 04:02:26.235198 4361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65fd18b3299744d8b7a7d3e7e455f17b [sys.catalog]: This master's current role is: LEADER
I20260927 04:02:26.235216 4362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65fd18b3299744d8b7a7d3e7e455f17b [sys.catalog]: This master's current role is: LEADER
I20260927 04:02:26.235504 4370 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:02:26.236116 4370 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:02:26.238016 4370 catalog_manager.cc:1347] Generated new cluster ID: 448e2f65281c474c93380b3a3601ac27
I20260927 04:02:26.238104 4370 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:02:26.238242 4360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:26.231977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50107 (local address 127.0.0.1:38973)
0927 04:02:26.232201 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:26.232212 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:26.232251 (+ 39us) server_negotiation.cc:407] Connection header received
0927 04:02:26.232601 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:26.232611 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:26.232679 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:26.232835 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:26.233082 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:26.234475 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:26.235622 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:26.236336 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:26.236442 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:26.237446 (+ 1004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:26.237450 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:26.237450 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:02:26.237679 (+ 229us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:26.237718 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:02:26.237733 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:26.237812 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:26.237928 (+ 116us) server_negotiation.cc:299] Negotiation successful
0927 04:02:26.237992 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":91,"threads_started":1}
I20260927 04:02:26.239199 16101 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 04:02:26.246656 4360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:26.240754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41289 (local address 127.0.0.1:38973)
0927 04:02:26.240792 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:26.240794 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:26.243293 (+ 2499us) server_negotiation.cc:407] Connection header received
0927 04:02:26.243345 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:26.243346 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:26.243358 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:26.243420 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:26.243555 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:26.244275 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:26.245319 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:26.245608 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:26.245667 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:26.246406 (+ 739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:26.246408 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:26.246408 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:02:26.246414 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:26.246428 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:02:26.246433 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:26.246481 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:26.246551 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 04:02:26.246573 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 04:02:26.264447 4370 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:02:26.265218 4370 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:02:26.279070 4370 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65fd18b3299744d8b7a7d3e7e455f17b: Generated new TSK 0
I20260927 04:02:26.279313 4370 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 04:02:26.290249 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 4288
2026-09-27T04:02:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1358 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481498018407-16101-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:02:26 dist-test-slave-glgz krb5kdc[4387](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:02:26 dist-test-slave-glgz krb5kdc[4387](info): set up 2 sockets
Sep 27 04:02:26 dist-test-slave-glgz krb5kdc[4387](info): commencing operation
krb5kdc: starting...
W20260927 04:02:27.335726 16101 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Sep 27 04:02:27 dist-test-slave-glgz krb5kdc[4387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:02:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:02:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481498018407-16101-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481498018407-16101-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 04:02:27.455981 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481498018407-16101-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481498018407-16101-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481498018407-16101-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42879
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481498018407-16101-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:02:27.537578 4400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:02:27.537799 4400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:02:27.537823 4400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:02:27.539084 4400 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:02:27.539142 4400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:02:27.539158 4400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:02:27.539212 4400 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:02:27.539229 4400 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:02:27.540859 4400 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481498018407-16101-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481498018407-16101-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481498018407-16101-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481498018407-16101-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:02:27.541111 4400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:02:27.541347 4400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:02:27.546445 4408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:27.546448 4405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:27.546476 4406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:27.546823 4400 server_base.cc:1034] running on GCE node
I20260927 04:02:27.547092 4400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:02:27.547375 4400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:02:27.548516 4400 hybrid_clock.cc:648] HybridClock initialized: now 1790481747548504 us; error 31 us; skew 500 ppm
Sep 27 04:02:27 dist-test-slave-glgz krb5kdc[4387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481747, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:02:27.550968 4400 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 04:02:27.551429 4400 webserver.cc:460] Webserver started at http://127.0.0.1:35607/ using document root <none> and password file <none>
I20260927 04:02:27.551666 4400 fs_manager.cc:359] Metadata directory not provided
I20260927 04:02:27.551734 4400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:02:27.551885 4400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:02:27.552937 4400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481498018407-16101-0/minicluster-data/master-0/data/instance:
uuid: "8608b5b5c3514d1b9ceff376644d6f62"
format_stamp: "Formatted at 2026-09-27 04:02:27 on dist-test-slave-glgz"
server_key: "4943a863915c9d2b3cb74d16f9c94b37"
server_key_iv: "dcf0165187fa72b8a3edb9fb934cd4b6"
server_key_version: "encryptionkey@0"
I20260927 04:02:27.553303 4400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481498018407-16101-0/minicluster-data/master-0/wal/instance:
uuid: "8608b5b5c3514d1b9ceff376644d6f62"
format_stamp: "Formatted at 2026-09-27 04:02:27 on dist-test-slave-glgz"
server_key: "4943a863915c9d2b3cb74d16f9c94b37"
server_key_iv: "dcf0165187fa72b8a3edb9fb934cd4b6"
server_key_version: "encryptionkey@0"
I20260927 04:02:27.554723 4400 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 04:02:27.555557 4415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:27.555738 4400 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:27.555841 4400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481498018407-16101-0/minicluster-data/master-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481498018407-16101-0/minicluster-data/master-0/wal
uuid: "8608b5b5c3514d1b9ceff376644d6f62"
format_stamp: "Formatted at 2026-09-27 04:02:27 on dist-test-slave-glgz"
server_key: "4943a863915c9d2b3cb74d16f9c94b37"
server_key_iv: "dcf0165187fa72b8a3edb9fb934cd4b6"
server_key_version: "encryptionkey@0"
I20260927 04:02:27.555927 4400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481498018407-16101-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481498018407-16101-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481498018407-16101-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:02:27.559303 4400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:02:27.561283 4400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:02:27.561451 4400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:02:27.565387 4467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42879 every 8 connection(s)
I20260927 04:02:27.565398 4400 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42879
I20260927 04:02:27.565797 4400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
I20260927 04:02:27.566731 4468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:02:27.569131 4468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8608b5b5c3514d1b9ceff376644d6f62: Bootstrap starting.
I20260927 04:02:27.569732 4468 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8608b5b5c3514d1b9ceff376644d6f62: Neither blocks nor log segments found. Creating new log.
I20260927 04:02:27.570022 4468 log.cc:824] T 00000000000000000000000000000000 P 8608b5b5c3514d1b9ceff376644d6f62: Log is configured to *not* fsync() on all Append() calls
I20260927 04:02:27.570761 4468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8608b5b5c3514d1b9ceff376644d6f62: No bootstrap required, opened a new log
I20260927 04:02:27.571009 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 4400
I20260927 04:02:27.571170 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481498018407-16101-0/minicluster-data/master-0/wal/instance
I20260927 04:02:27.571379 16101 external_mini_cluster.cc:1428] Setting key 63698249bb76b701169d673cd3e3611d
I20260927 04:02:27.572930 4468 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8608b5b5c3514d1b9ceff376644d6f62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8608b5b5c3514d1b9ceff376644d6f62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42879 } }
I20260927 04:02:27.573133 4468 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8608b5b5c3514d1b9ceff376644d6f62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:02:27.573176 4468 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8608b5b5c3514d1b9ceff376644d6f62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8608b5b5c3514d1b9ceff376644d6f62, State: Initialized, Role: FOLLOWER
I20260927 04:02:27.573334 4468 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8608b5b5c3514d1b9ceff376644d6f62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8608b5b5c3514d1b9ceff376644d6f62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42879 } }
I20260927 04:02:27.573807 4468 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8608b5b5c3514d1b9ceff376644d6f62 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:02:27.573877 4468 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8608b5b5c3514d1b9ceff376644d6f62 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:02:27.573930 4468 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8608b5b5c3514d1b9ceff376644d6f62 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:02:27.573977 4472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:27.571911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52201 (local address 127.0.0.1:42879)
0927 04:02:27.572133 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:27.572143 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:27.572178 (+ 35us) server_negotiation.cc:407] Connection header received
0927 04:02:27.572564 (+ 386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:27.572571 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:27.572596 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:27.572752 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:27.573471 (+ 719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:27.573476 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:27.573476 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:02:27.573480 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:27.573537 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:02:27.573547 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:27.573627 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:27.573698 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 04:02:27.573739 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":86,"threads_started":1}
I20260927 04:02:27.574594 4468 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8608b5b5c3514d1b9ceff376644d6f62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8608b5b5c3514d1b9ceff376644d6f62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42879 } }
I20260927 04:02:27.574731 4468 leader_election.cc:302] T 00000000000000000000000000000000 P 8608b5b5c3514d1b9ceff376644d6f62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8608b5b5c3514d1b9ceff376644d6f62; no voters:
I20260927 04:02:27.574905 4468 leader_election.cc:288] T 00000000000000000000000000000000 P 8608b5b5c3514d1b9ceff376644d6f62 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:02:27.574990 16101 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 04:02:27.574978 4473 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8608b5b5c3514d1b9ceff376644d6f62 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:02:27.575153 4473 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8608b5b5c3514d1b9ceff376644d6f62 [term 1 LEADER]: Becoming Leader. State: Replica: 8608b5b5c3514d1b9ceff376644d6f62, State: Running, Role: LEADER
I20260927 04:02:27.575222 4468 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8608b5b5c3514d1b9ceff376644d6f62 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:02:27.575292 4473 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8608b5b5c3514d1b9ceff376644d6f62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8608b5b5c3514d1b9ceff376644d6f62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42879 } }
I20260927 04:02:27.575685 4473 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8608b5b5c3514d1b9ceff376644d6f62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8608b5b5c3514d1b9ceff376644d6f62. Latest consensus state: current_term: 1 leader_uuid: "8608b5b5c3514d1b9ceff376644d6f62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8608b5b5c3514d1b9ceff376644d6f62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42879 } } }
I20260927 04:02:27.575696 4474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8608b5b5c3514d1b9ceff376644d6f62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8608b5b5c3514d1b9ceff376644d6f62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8608b5b5c3514d1b9ceff376644d6f62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42879 } } }
I20260927 04:02:27.575960 4474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8608b5b5c3514d1b9ceff376644d6f62 [sys.catalog]: This master's current role is: LEADER
I20260927 04:02:27.576179 4473 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8608b5b5c3514d1b9ceff376644d6f62 [sys.catalog]: This master's current role is: LEADER
W20260927 04:02:27.577030 4493 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8608b5b5c3514d1b9ceff376644d6f62: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 04:02:27.577116 4493 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 04:02:27.577171 4482 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:02:27.577324 4482 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:02:27.579089 4482 catalog_manager.cc:1347] Generated new cluster ID: 019a3e85c0c14570b18820d3849f07dd
I20260927 04:02:27.579140 4482 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:02:27.583379 4472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:27.577025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34425 (local address 127.0.0.1:42879)
0927 04:02:27.577061 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:27.577063 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:27.578106 (+ 1043us) server_negotiation.cc:407] Connection header received
0927 04:02:27.578182 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:27.578183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:27.578189 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:27.578257 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:27.578939 (+ 682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:27.578941 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:27.578941 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:02:27.578945 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:27.578959 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:02:27.578964 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:27.578993 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:27.583267 (+ 4274us) server_negotiation.cc:299] Negotiation successful
0927 04:02:27.583286 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260927 04:02:27.600857 4482 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:02:27.601080 4482 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:02:27.606159 4482 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8608b5b5c3514d1b9ceff376644d6f62: Generated new TSK 0
I20260927 04:02:27.606372 4482 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 04:02:27.612644 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 4400
2026-09-27T04:02:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1322 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481498018407-16101-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:02:27 dist-test-slave-glgz krb5kdc[4499](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:02:27 dist-test-slave-glgz krb5kdc[4499](info): set up 2 sockets
Sep 27 04:02:27 dist-test-slave-glgz krb5kdc[4499](info): commencing operation
krb5kdc: starting...
W20260927 04:02:28.655671 16101 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Sep 27 04:02:28 dist-test-slave-glgz krb5kdc[4499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:02:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:02:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481498018407-16101-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481498018407-16101-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 04:02:28.777413 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481498018407-16101-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481498018407-16101-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481498018407-16101-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32817
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481498018407-16101-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32817
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:02:28.858376 4512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:02:28.858628 4512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:02:28.858669 4512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:02:28.859840 4512 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:02:28.859910 4512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:02:28.859933 4512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:02:28.859962 4512 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:02:28.859992 4512 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:02:28.861476 4512 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481498018407-16101-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481498018407-16101-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32817
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481498018407-16101-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481498018407-16101-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:02:28.861742 4512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:02:28.861965 4512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:02:28.866850 4518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:28.866823 4517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:28.866835 4520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:28.867079 4512 server_base.cc:1034] running on GCE node
I20260927 04:02:28.867442 4512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:02:28.867754 4512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:02:28.868947 4512 hybrid_clock.cc:648] HybridClock initialized: now 1790481748868940 us; error 44 us; skew 500 ppm
Sep 27 04:02:28 dist-test-slave-glgz krb5kdc[4499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481748, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:02:28.871755 4512 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 04:02:28.872216 4512 webserver.cc:460] Webserver started at http://127.0.0.1:42853/ using document root <none> and password file <none>
I20260927 04:02:28.872432 4512 fs_manager.cc:359] Metadata directory not provided
I20260927 04:02:28.872484 4512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:02:28.872594 4512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:02:28.873781 4512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481498018407-16101-0/minicluster-data/master-0/data/instance:
uuid: "af9ab510de86497db3804c95566ff788"
format_stamp: "Formatted at 2026-09-27 04:02:28 on dist-test-slave-glgz"
server_key: "82eed387f1e9fcd1de919f9bcc7d7f1f"
server_key_iv: "a9e9b7e39ebc3bd76b0447c003566a4b"
server_key_version: "encryptionkey@0"
I20260927 04:02:28.874109 4512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481498018407-16101-0/minicluster-data/master-0/wal/instance:
uuid: "af9ab510de86497db3804c95566ff788"
format_stamp: "Formatted at 2026-09-27 04:02:28 on dist-test-slave-glgz"
server_key: "82eed387f1e9fcd1de919f9bcc7d7f1f"
server_key_iv: "a9e9b7e39ebc3bd76b0447c003566a4b"
server_key_version: "encryptionkey@0"
I20260927 04:02:28.875491 4512 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 04:02:28.876271 4527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:28.876457 4512 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:28.876525 4512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481498018407-16101-0/minicluster-data/master-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481498018407-16101-0/minicluster-data/master-0/wal
uuid: "af9ab510de86497db3804c95566ff788"
format_stamp: "Formatted at 2026-09-27 04:02:28 on dist-test-slave-glgz"
server_key: "82eed387f1e9fcd1de919f9bcc7d7f1f"
server_key_iv: "a9e9b7e39ebc3bd76b0447c003566a4b"
server_key_version: "encryptionkey@0"
I20260927 04:02:28.876585 4512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481498018407-16101-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481498018407-16101-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481498018407-16101-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:02:28.908349 4512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:02:28.910341 4512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:02:28.910451 4512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:02:28.914386 4512 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32817
I20260927 04:02:28.914448 4579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32817 every 8 connection(s)
I20260927 04:02:28.914810 4512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
I20260927 04:02:28.915707 4580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:02:28.918007 4580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af9ab510de86497db3804c95566ff788: Bootstrap starting.
I20260927 04:02:28.918604 4580 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af9ab510de86497db3804c95566ff788: Neither blocks nor log segments found. Creating new log.
I20260927 04:02:28.918895 4580 log.cc:824] T 00000000000000000000000000000000 P af9ab510de86497db3804c95566ff788: Log is configured to *not* fsync() on all Append() calls
I20260927 04:02:28.919657 4580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af9ab510de86497db3804c95566ff788: No bootstrap required, opened a new log
I20260927 04:02:28.921248 4580 raft_consensus.cc:348] T 00000000000000000000000000000000 P af9ab510de86497db3804c95566ff788 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af9ab510de86497db3804c95566ff788" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32817 } }
I20260927 04:02:28.921414 4580 raft_consensus.cc:374] T 00000000000000000000000000000000 P af9ab510de86497db3804c95566ff788 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:02:28.921459 4580 raft_consensus.cc:729] T 00000000000000000000000000000000 P af9ab510de86497db3804c95566ff788 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af9ab510de86497db3804c95566ff788, State: Initialized, Role: FOLLOWER
I20260927 04:02:28.921628 4580 consensus_queue.cc:260] T 00000000000000000000000000000000 P af9ab510de86497db3804c95566ff788 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af9ab510de86497db3804c95566ff788" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32817 } }
I20260927 04:02:28.921727 4580 raft_consensus.cc:388] T 00000000000000000000000000000000 P af9ab510de86497db3804c95566ff788 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:02:28.921772 4580 raft_consensus.cc:482] T 00000000000000000000000000000000 P af9ab510de86497db3804c95566ff788 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:02:28.921821 4580 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af9ab510de86497db3804c95566ff788 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:02:28.922356 4580 raft_consensus.cc:504] T 00000000000000000000000000000000 P af9ab510de86497db3804c95566ff788 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af9ab510de86497db3804c95566ff788" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32817 } }
I20260927 04:02:28.922473 4580 leader_election.cc:302] T 00000000000000000000000000000000 P af9ab510de86497db3804c95566ff788 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af9ab510de86497db3804c95566ff788; no voters:
I20260927 04:02:28.922608 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 4512
I20260927 04:02:28.922670 4580 leader_election.cc:288] T 00000000000000000000000000000000 P af9ab510de86497db3804c95566ff788 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:02:28.922691 4583 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af9ab510de86497db3804c95566ff788 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:02:28.922763 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481498018407-16101-0/minicluster-data/master-0/wal/instance
I20260927 04:02:28.922806 4583 raft_consensus.cc:686] T 00000000000000000000000000000000 P af9ab510de86497db3804c95566ff788 [term 1 LEADER]: Becoming Leader. State: Replica: af9ab510de86497db3804c95566ff788, State: Running, Role: LEADER
I20260927 04:02:28.922890 4583 consensus_queue.cc:237] T 00000000000000000000000000000000 P af9ab510de86497db3804c95566ff788 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af9ab510de86497db3804c95566ff788" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32817 } }
I20260927 04:02:28.922935 4580 sys_catalog.cc:564] T 00000000000000000000000000000000 P af9ab510de86497db3804c95566ff788 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:02:28.922984 16101 external_mini_cluster.cc:1428] Setting key a8c4f9addbc3d6fbf4bbb5b1e6575535
I20260927 04:02:28.923393 4585 sys_catalog.cc:455] T 00000000000000000000000000000000 P af9ab510de86497db3804c95566ff788 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af9ab510de86497db3804c95566ff788. Latest consensus state: current_term: 1 leader_uuid: "af9ab510de86497db3804c95566ff788" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af9ab510de86497db3804c95566ff788" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32817 } } }
I20260927 04:02:28.923543 4585 sys_catalog.cc:458] T 00000000000000000000000000000000 P af9ab510de86497db3804c95566ff788 [sys.catalog]: This master's current role is: LEADER
I20260927 04:02:28.923247 4584 sys_catalog.cc:455] T 00000000000000000000000000000000 P af9ab510de86497db3804c95566ff788 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af9ab510de86497db3804c95566ff788" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af9ab510de86497db3804c95566ff788" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32817 } } }
I20260927 04:02:28.923763 4584 sys_catalog.cc:458] T 00000000000000000000000000000000 P af9ab510de86497db3804c95566ff788 [sys.catalog]: This master's current role is: LEADER
I20260927 04:02:28.924054 4592 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:02:28.924654 4592 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:02:28.926470 4592 catalog_manager.cc:1347] Generated new cluster ID: 61188d5d65c243839a3e344a0f1d3439
I20260927 04:02:28.926530 4592 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 04:02:28 dist-test-slave-glgz krb5kdc[4499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 04:02:28.935189 4588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:28.923510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.151:40959 (local address 127.0.0.1:32817)
0927 04:02:28.923797 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:28.923809 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:28.923849 (+ 40us) server_negotiation.cc:407] Connection header received
0927 04:02:28.923975 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:28.923984 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:28.924050 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:28.925026 (+ 976us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:28.925147 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:28.926407 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:28.927488 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:28.928097 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:28.928205 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:28.931089 (+ 2884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:28.931093 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:28.931093 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:28.931105 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:28.933344 (+ 2239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:28.933845 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:28.933849 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:28.933849 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:28.933983 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:28.934231 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:28.934232 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:28.934232 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:28.934535 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:28.934692 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:28.934834 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 04:02:28.934931 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":71,"threads_started":1}
I20260927 04:02:28.936172 16101 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 04:02:28.944972 4588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:28.937635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.151:56657 (local address 127.0.0.1:32817)
0927 04:02:28.937671 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:28.937673 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:28.937803 (+ 130us) server_negotiation.cc:407] Connection header received
0927 04:02:28.937833 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:28.937834 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:28.937844 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:28.937912 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:28.938486 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:28.939316 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:28.940289 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:28.940641 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:28.940713 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:28.942056 (+ 1343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:28.942058 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:28.942058 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:28.942065 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:28.943735 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:28.944153 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:28.944154 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:28.944154 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:28.944217 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:28.944439 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:28.944439 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:28.944440 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:28.944583 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:28.944689 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:28.944806 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 04:02:28.944872 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 04:02:28.949563 4592 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:02:28.950289 4592 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:02:28.955897 4592 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af9ab510de86497db3804c95566ff788: Generated new TSK 0
I20260927 04:02:28.956125 4592 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 04:02:28.960131 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 4512
W20260927 04:02:28.964277 4497 connection.cc:535] client connection to 127.0.0.1:32817 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32817): Connection reset by peer (error 104)
2026-09-27T04:02:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1346 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481498018407-16101-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:02:29 dist-test-slave-glgz krb5kdc[4611](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:02:29 dist-test-slave-glgz krb5kdc[4611](info): set up 2 sockets
Sep 27 04:02:29 dist-test-slave-glgz krb5kdc[4611](info): commencing operation
krb5kdc: starting...
W20260927 04:02:29.991930 16101 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Sep 27 04:02:30 dist-test-slave-glgz krb5kdc[4611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:02:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:02:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481498018407-16101-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481498018407-16101-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 04:02:30.113770 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481498018407-16101-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481498018407-16101-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481498018407-16101-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41811
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481498018407-16101-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41811
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:02:30.195935 4624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:02:30.196137 4624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:02:30.196161 4624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:02:30.197252 4624 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:02:30.197288 4624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:02:30.197302 4624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:02:30.197314 4624 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:02:30.197326 4624 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:02:30.198732 4624 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481498018407-16101-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481498018407-16101-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41811
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481498018407-16101-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481498018407-16101-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:02:30.198923 4624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:02:30.199165 4624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:02:30.203820 4629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:30.203938 4630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:30.203850 4632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:30.204228 4624 server_base.cc:1034] running on GCE node
I20260927 04:02:30.204475 4624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:02:30.204805 4624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:02:30.205992 4624 hybrid_clock.cc:648] HybridClock initialized: now 1790481750205972 us; error 39 us; skew 500 ppm
Sep 27 04:02:30 dist-test-slave-glgz krb5kdc[4611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481750, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:02:30.208678 4624 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 04:02:30.209105 4624 webserver.cc:460] Webserver started at http://127.0.0.1:41401/ using document root <none> and password file <none>
I20260927 04:02:30.209349 4624 fs_manager.cc:359] Metadata directory not provided
I20260927 04:02:30.209419 4624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:02:30.209563 4624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:02:30.210606 4624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481498018407-16101-0/minicluster-data/master-0/data/instance:
uuid: "5bc71d387a24401d8fc9b3839db024a1"
format_stamp: "Formatted at 2026-09-27 04:02:30 on dist-test-slave-glgz"
server_key: "f1313d959493961a9121e436b1002eb1"
server_key_iv: "29c0fb47a2b1342fd3b30d73f20b27b6"
server_key_version: "encryptionkey@0"
I20260927 04:02:30.210980 4624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481498018407-16101-0/minicluster-data/master-0/wal/instance:
uuid: "5bc71d387a24401d8fc9b3839db024a1"
format_stamp: "Formatted at 2026-09-27 04:02:30 on dist-test-slave-glgz"
server_key: "f1313d959493961a9121e436b1002eb1"
server_key_iv: "29c0fb47a2b1342fd3b30d73f20b27b6"
server_key_version: "encryptionkey@0"
I20260927 04:02:30.212358 4624 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:02:30.213166 4639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:30.213351 4624 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 04:02:30.213465 4624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481498018407-16101-0/minicluster-data/master-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481498018407-16101-0/minicluster-data/master-0/wal
uuid: "5bc71d387a24401d8fc9b3839db024a1"
format_stamp: "Formatted at 2026-09-27 04:02:30 on dist-test-slave-glgz"
server_key: "f1313d959493961a9121e436b1002eb1"
server_key_iv: "29c0fb47a2b1342fd3b30d73f20b27b6"
server_key_version: "encryptionkey@0"
I20260927 04:02:30.213554 4624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481498018407-16101-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481498018407-16101-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481498018407-16101-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:02:30.228327 4624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:02:30.230389 4624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:02:30.230574 4624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:02:30.234835 4624 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41811
I20260927 04:02:30.234879 4691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41811 every 8 connection(s)
I20260927 04:02:30.235309 4624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
I20260927 04:02:30.236263 4692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:02:30.238608 4692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bc71d387a24401d8fc9b3839db024a1: Bootstrap starting.
I20260927 04:02:30.238930 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 4624
I20260927 04:02:30.239104 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481498018407-16101-0/minicluster-data/master-0/wal/instance
I20260927 04:02:30.239310 16101 external_mini_cluster.cc:1428] Setting key db1b17bfbeb9bc30bb0bce1c9b2a049b
I20260927 04:02:30.239305 4692 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5bc71d387a24401d8fc9b3839db024a1: Neither blocks nor log segments found. Creating new log.
I20260927 04:02:30.239603 4692 log.cc:824] T 00000000000000000000000000000000 P 5bc71d387a24401d8fc9b3839db024a1: Log is configured to *not* fsync() on all Append() calls
I20260927 04:02:30.240451 4692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bc71d387a24401d8fc9b3839db024a1: No bootstrap required, opened a new log
I20260927 04:02:30.242848 4692 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5bc71d387a24401d8fc9b3839db024a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bc71d387a24401d8fc9b3839db024a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41811 } }
I20260927 04:02:30.243013 4692 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5bc71d387a24401d8fc9b3839db024a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:02:30.243084 4692 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5bc71d387a24401d8fc9b3839db024a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bc71d387a24401d8fc9b3839db024a1, State: Initialized, Role: FOLLOWER
I20260927 04:02:30.243247 4692 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bc71d387a24401d8fc9b3839db024a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bc71d387a24401d8fc9b3839db024a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41811 } }
I20260927 04:02:30.243343 4692 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5bc71d387a24401d8fc9b3839db024a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:02:30.243402 4692 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5bc71d387a24401d8fc9b3839db024a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:02:30.243454 4692 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5bc71d387a24401d8fc9b3839db024a1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:02:30.243981 4692 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5bc71d387a24401d8fc9b3839db024a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bc71d387a24401d8fc9b3839db024a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41811 } }
I20260927 04:02:30.244108 4692 leader_election.cc:302] T 00000000000000000000000000000000 P 5bc71d387a24401d8fc9b3839db024a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5bc71d387a24401d8fc9b3839db024a1; no voters:
I20260927 04:02:30.244297 4692 leader_election.cc:288] T 00000000000000000000000000000000 P 5bc71d387a24401d8fc9b3839db024a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:02:30.244323 4697 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5bc71d387a24401d8fc9b3839db024a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:02:30.244511 4697 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5bc71d387a24401d8fc9b3839db024a1 [term 1 LEADER]: Becoming Leader. State: Replica: 5bc71d387a24401d8fc9b3839db024a1, State: Running, Role: LEADER
I20260927 04:02:30.244545 4692 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5bc71d387a24401d8fc9b3839db024a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:02:30.244668 4697 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5bc71d387a24401d8fc9b3839db024a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bc71d387a24401d8fc9b3839db024a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41811 } }
I20260927 04:02:30.245497 4696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:30.239818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.151:33205 (local address 127.0.0.1:41811)
0927 04:02:30.239994 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:30.240002 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:30.240027 (+ 25us) server_negotiation.cc:407] Connection header received
0927 04:02:30.240655 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:30.240664 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:30.240727 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:30.240883 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:30.241050 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:30.242490 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:30.243403 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:30.243892 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:30.244020 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:30.244730 (+ 710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:30.244734 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:30.244734 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:02:30.244974 (+ 240us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:30.245018 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:02:30.245030 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:30.245119 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:30.245276 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 04:02:30.245308 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":88,"threads_started":1}
I20260927 04:02:30.245901 4698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bc71d387a24401d8fc9b3839db024a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bc71d387a24401d8fc9b3839db024a1. Latest consensus state: current_term: 1 leader_uuid: "5bc71d387a24401d8fc9b3839db024a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bc71d387a24401d8fc9b3839db024a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41811 } } }
I20260927 04:02:30.245990 4698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bc71d387a24401d8fc9b3839db024a1 [sys.catalog]: This master's current role is: LEADER
I20260927 04:02:30.246014 4699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bc71d387a24401d8fc9b3839db024a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5bc71d387a24401d8fc9b3839db024a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bc71d387a24401d8fc9b3839db024a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41811 } } }
I20260927 04:02:30.246085 4699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bc71d387a24401d8fc9b3839db024a1 [sys.catalog]: This master's current role is: LEADER
I20260927 04:02:30.246363 4709 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:02:30.246479 16101 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 04:02:30.246922 4709 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:02:30.248373 4709 catalog_manager.cc:1347] Generated new cluster ID: e8c3ea8a3c064dcc8d7b4eeaf9e37d3a
I20260927 04:02:30.248422 4709 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:02:30.254061 4696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:30.248042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.151:49573 (local address 127.0.0.1:41811)
0927 04:02:30.248076 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:30.248077 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:30.251330 (+ 3253us) server_negotiation.cc:407] Connection header received
0927 04:02:30.251396 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:30.251397 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:30.251409 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:30.251464 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:30.251613 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:30.252086 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:30.252815 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:30.253019 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:30.253101 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:30.253818 (+ 717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:30.253819 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:30.253820 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:02:30.253824 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:30.253839 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:02:30.253843 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:30.253910 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:30.253976 (+ 66us) server_negotiation.cc:299] Negotiation successful
0927 04:02:30.253984 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260927 04:02:30.265914 4709 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:02:30.266938 4709 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:02:30.275665 4709 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5bc71d387a24401d8fc9b3839db024a1: Generated new TSK 0
I20260927 04:02:30.275890 4709 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 04:02:30.287608 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 4624
W20260927 04:02:30.293743 4609 connection.cc:535] client connection to 127.0.0.1:41811 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41811): Connection reset by peer (error 104)
2026-09-27T04:02:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1329 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481498018407-16101-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:02:30 dist-test-slave-glgz krb5kdc[4723](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:02:30 dist-test-slave-glgz krb5kdc[4723](info): set up 2 sockets
Sep 27 04:02:30 dist-test-slave-glgz krb5kdc[4723](info): commencing operation
krb5kdc: starting...
W20260927 04:02:31.328104 16101 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Sep 27 04:02:31 dist-test-slave-glgz krb5kdc[4723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:02:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:02:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481498018407-16101-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481498018407-16101-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 04:02:31.449815 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481498018407-16101-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481498018407-16101-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481498018407-16101-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45655
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481498018407-16101-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45655
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:02:31.531180 4736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:02:31.531388 4736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:02:31.531412 4736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:02:31.532549 4736 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:02:31.532588 4736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:02:31.532609 4736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:02:31.532622 4736 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:02:31.532634 4736 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:02:31.534137 4736 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481498018407-16101-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481498018407-16101-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45655
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481498018407-16101-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481498018407-16101-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:02:31.534338 4736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:02:31.534509 4736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:02:31.539487 4744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:31.539583 4742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:31.539475 4741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:31.539703 4736 server_base.cc:1034] running on GCE node
I20260927 04:02:31.539956 4736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:02:31.540345 4736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:02:31.541499 4736 hybrid_clock.cc:648] HybridClock initialized: now 1790481751541481 us; error 41 us; skew 500 ppm
Sep 27 04:02:31 dist-test-slave-glgz krb5kdc[4723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481751, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:02:31.550981 4736 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 04:02:31.551561 4736 webserver.cc:460] Webserver started at http://127.0.0.1:46479/ using document root <none> and password file <none>
I20260927 04:02:31.551797 4736 fs_manager.cc:359] Metadata directory not provided
I20260927 04:02:31.551843 4736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:02:31.551944 4736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:02:31.553069 4736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481498018407-16101-0/minicluster-data/master-0/data/instance:
uuid: "1bff871e76ed401caa0d69199e708ecf"
format_stamp: "Formatted at 2026-09-27 04:02:31 on dist-test-slave-glgz"
server_key: "27693e3a7f903ee26b21788ada25e3d0"
server_key_iv: "5803077ad59364eee14b7af7660bbcdf"
server_key_version: "encryptionkey@0"
I20260927 04:02:31.553416 4736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481498018407-16101-0/minicluster-data/master-0/wal/instance:
uuid: "1bff871e76ed401caa0d69199e708ecf"
format_stamp: "Formatted at 2026-09-27 04:02:31 on dist-test-slave-glgz"
server_key: "27693e3a7f903ee26b21788ada25e3d0"
server_key_iv: "5803077ad59364eee14b7af7660bbcdf"
server_key_version: "encryptionkey@0"
I20260927 04:02:31.554905 4736 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 04:02:31.555789 4751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:31.556003 4736 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:02:31.556085 4736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481498018407-16101-0/minicluster-data/master-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481498018407-16101-0/minicluster-data/master-0/wal
uuid: "1bff871e76ed401caa0d69199e708ecf"
format_stamp: "Formatted at 2026-09-27 04:02:31 on dist-test-slave-glgz"
server_key: "27693e3a7f903ee26b21788ada25e3d0"
server_key_iv: "5803077ad59364eee14b7af7660bbcdf"
server_key_version: "encryptionkey@0"
I20260927 04:02:31.556164 4736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481498018407-16101-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481498018407-16101-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481498018407-16101-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:02:31.559484 4736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:02:31.561311 4736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:02:31.561425 4736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:02:31.565500 4736 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45655
I20260927 04:02:31.565536 4803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45655 every 8 connection(s)
I20260927 04:02:31.565898 4736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
I20260927 04:02:31.566804 4804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:02:31.569509 4804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bff871e76ed401caa0d69199e708ecf: Bootstrap starting.
I20260927 04:02:31.570194 4804 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1bff871e76ed401caa0d69199e708ecf: Neither blocks nor log segments found. Creating new log.
I20260927 04:02:31.570508 4804 log.cc:824] T 00000000000000000000000000000000 P 1bff871e76ed401caa0d69199e708ecf: Log is configured to *not* fsync() on all Append() calls
I20260927 04:02:31.571344 4804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bff871e76ed401caa0d69199e708ecf: No bootstrap required, opened a new log
I20260927 04:02:31.573076 4804 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1bff871e76ed401caa0d69199e708ecf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bff871e76ed401caa0d69199e708ecf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45655 } }
I20260927 04:02:31.573230 4804 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1bff871e76ed401caa0d69199e708ecf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:02:31.573253 4804 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1bff871e76ed401caa0d69199e708ecf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bff871e76ed401caa0d69199e708ecf, State: Initialized, Role: FOLLOWER
I20260927 04:02:31.573460 4804 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1bff871e76ed401caa0d69199e708ecf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bff871e76ed401caa0d69199e708ecf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45655 } }
I20260927 04:02:31.573563 4804 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1bff871e76ed401caa0d69199e708ecf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:02:31.573590 4804 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1bff871e76ed401caa0d69199e708ecf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:02:31.573645 4804 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1bff871e76ed401caa0d69199e708ecf [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:02:31.574242 4804 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1bff871e76ed401caa0d69199e708ecf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bff871e76ed401caa0d69199e708ecf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45655 } }
I20260927 04:02:31.574361 4804 leader_election.cc:302] T 00000000000000000000000000000000 P 1bff871e76ed401caa0d69199e708ecf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1bff871e76ed401caa0d69199e708ecf; no voters:
I20260927 04:02:31.574571 4804 leader_election.cc:288] T 00000000000000000000000000000000 P 1bff871e76ed401caa0d69199e708ecf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:02:31.574620 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 4736
I20260927 04:02:31.574627 4807 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1bff871e76ed401caa0d69199e708ecf [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:02:31.574756 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481498018407-16101-0/minicluster-data/master-0/wal/instance
I20260927 04:02:31.574801 4807 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1bff871e76ed401caa0d69199e708ecf [term 1 LEADER]: Becoming Leader. State: Replica: 1bff871e76ed401caa0d69199e708ecf, State: Running, Role: LEADER
I20260927 04:02:31.574872 4804 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1bff871e76ed401caa0d69199e708ecf [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:02:31.574908 4807 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1bff871e76ed401caa0d69199e708ecf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bff871e76ed401caa0d69199e708ecf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45655 } }
I20260927 04:02:31.574975 16101 external_mini_cluster.cc:1428] Setting key 0d43141055ba14c8410b52a0f00fc9fa
I20260927 04:02:31.575330 4807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bff871e76ed401caa0d69199e708ecf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bff871e76ed401caa0d69199e708ecf. Latest consensus state: current_term: 1 leader_uuid: "1bff871e76ed401caa0d69199e708ecf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bff871e76ed401caa0d69199e708ecf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45655 } } }
I20260927 04:02:31.575428 4807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bff871e76ed401caa0d69199e708ecf [sys.catalog]: This master's current role is: LEADER
I20260927 04:02:31.575340 4808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bff871e76ed401caa0d69199e708ecf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1bff871e76ed401caa0d69199e708ecf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bff871e76ed401caa0d69199e708ecf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45655 } } }
I20260927 04:02:31.575629 4808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bff871e76ed401caa0d69199e708ecf [sys.catalog]: This master's current role is: LEADER
I20260927 04:02:31.575987 4812 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:02:31.576666 4812 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:02:31.578166 4813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:31.575670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.151:36811 (local address 127.0.0.1:45655)
0927 04:02:31.575843 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:31.575853 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:31.575894 (+ 41us) server_negotiation.cc:407] Connection header received
0927 04:02:31.576194 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:31.576204 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:31.576241 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:31.576400 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:31.577544 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:31.577548 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:31.577549 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:02:31.577553 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:31.577592 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:02:31.577612 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:31.577686 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:31.577933 (+ 247us) server_negotiation.cc:299] Negotiation successful
0927 04:02:31.577986 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":64,"threads_started":1}
I20260927 04:02:31.579275 16101 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 04:02:31.579376 4812 catalog_manager.cc:1347] Generated new cluster ID: 5119ccab9adf4695a22c0c6b4790bf47
I20260927 04:02:31.579430 4812 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:02:31.584843 4813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:31.583372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.151:55019 (local address 127.0.0.1:45655)
0927 04:02:31.583432 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:31.583434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:31.583607 (+ 173us) server_negotiation.cc:407] Connection header received
0927 04:02:31.583777 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:31.583778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:31.583784 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:31.583814 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:31.584543 (+ 729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:31.584545 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:31.584545 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:02:31.584550 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:31.584571 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:02:31.584577 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:31.584659 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:31.584714 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 04:02:31.584733 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 04:02:31.602283 4812 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:02:31.602526 4812 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:02:31.607980 4812 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1bff871e76ed401caa0d69199e708ecf: Generated new TSK 0
I20260927 04:02:31.608225 4812 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 04:02:31.614749 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 4736
2026-09-27T04:02:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1325 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481498018407-16101-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:02:31 dist-test-slave-glgz krb5kdc[4835](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:02:31 dist-test-slave-glgz krb5kdc[4835](info): set up 2 sockets
Sep 27 04:02:31 dist-test-slave-glgz krb5kdc[4835](info): commencing operation
krb5kdc: starting...
W20260927 04:02:32.731588 16101 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Sep 27 04:02:32 dist-test-slave-glgz krb5kdc[4835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:02:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:02:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481498018407-16101-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481498018407-16101-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 04:02:32.854647 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481498018407-16101-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481498018407-16101-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481498018407-16101-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38713
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481498018407-16101-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38713
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:02:32.936859 4848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:02:32.937129 4848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:02:32.937176 4848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:02:32.938361 4848 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:02:32.938432 4848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:02:32.938455 4848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:02:32.938478 4848 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:02:32.938506 4848 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:02:32.940026 4848 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481498018407-16101-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481498018407-16101-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38713
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481498018407-16101-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481498018407-16101-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:02:32.940323 4848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:02:32.940555 4848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:02:32.945497 4856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:32.945497 4854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:32.945716 4853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:32.946048 4848 server_base.cc:1034] running on GCE node
I20260927 04:02:32.946291 4848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:02:32.946579 4848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:02:32.947723 4848 hybrid_clock.cc:648] HybridClock initialized: now 1790481752947707 us; error 25 us; skew 500 ppm
Sep 27 04:02:32 dist-test-slave-glgz krb5kdc[4835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481752, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:02:32.950397 4848 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 04:02:32.950807 4848 webserver.cc:460] Webserver started at http://127.0.0.1:38165/ using document root <none> and password file <none>
I20260927 04:02:32.951082 4848 fs_manager.cc:359] Metadata directory not provided
I20260927 04:02:32.951150 4848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:02:32.951303 4848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:02:32.952435 4848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481498018407-16101-0/minicluster-data/master-0/data/instance:
uuid: "1da49ee0527f49c1b6a05b096a7feb5e"
format_stamp: "Formatted at 2026-09-27 04:02:32 on dist-test-slave-glgz"
server_key: "74b17aec9106d3d07792b35904826605"
server_key_iv: "2bf252c456591940d5bef5627e73544e"
server_key_version: "encryptionkey@0"
I20260927 04:02:32.952813 4848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481498018407-16101-0/minicluster-data/master-0/wal/instance:
uuid: "1da49ee0527f49c1b6a05b096a7feb5e"
format_stamp: "Formatted at 2026-09-27 04:02:32 on dist-test-slave-glgz"
server_key: "74b17aec9106d3d07792b35904826605"
server_key_iv: "2bf252c456591940d5bef5627e73544e"
server_key_version: "encryptionkey@0"
I20260927 04:02:32.954233 4848 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 04:02:32.955001 4863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:32.955245 4848 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:02:32.955339 4848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481498018407-16101-0/minicluster-data/master-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481498018407-16101-0/minicluster-data/master-0/wal
uuid: "1da49ee0527f49c1b6a05b096a7feb5e"
format_stamp: "Formatted at 2026-09-27 04:02:32 on dist-test-slave-glgz"
server_key: "74b17aec9106d3d07792b35904826605"
server_key_iv: "2bf252c456591940d5bef5627e73544e"
server_key_version: "encryptionkey@0"
I20260927 04:02:32.955415 4848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481498018407-16101-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481498018407-16101-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481498018407-16101-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:02:32.968078 4848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:02:32.970347 4848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:02:32.970512 4848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:02:32.974449 4848 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38713
I20260927 04:02:32.974500 4915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38713 every 8 connection(s)
I20260927 04:02:32.974843 4848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
I20260927 04:02:32.975829 4916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:02:32.978261 4916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1da49ee0527f49c1b6a05b096a7feb5e: Bootstrap starting.
I20260927 04:02:32.978819 4916 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1da49ee0527f49c1b6a05b096a7feb5e: Neither blocks nor log segments found. Creating new log.
I20260927 04:02:32.979131 4916 log.cc:824] T 00000000000000000000000000000000 P 1da49ee0527f49c1b6a05b096a7feb5e: Log is configured to *not* fsync() on all Append() calls
I20260927 04:02:32.979635 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 4848
I20260927 04:02:32.979787 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481498018407-16101-0/minicluster-data/master-0/wal/instance
I20260927 04:02:32.979900 4916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1da49ee0527f49c1b6a05b096a7feb5e: No bootstrap required, opened a new log
I20260927 04:02:32.980032 16101 external_mini_cluster.cc:1428] Setting key 5e9b50c6bb2cf9fa5db899732ea84c2f
I20260927 04:02:32.982252 4916 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1da49ee0527f49c1b6a05b096a7feb5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1da49ee0527f49c1b6a05b096a7feb5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38713 } }
I20260927 04:02:32.982462 4916 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1da49ee0527f49c1b6a05b096a7feb5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:02:32.982517 4916 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1da49ee0527f49c1b6a05b096a7feb5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1da49ee0527f49c1b6a05b096a7feb5e, State: Initialized, Role: FOLLOWER
I20260927 04:02:32.982679 4916 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1da49ee0527f49c1b6a05b096a7feb5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1da49ee0527f49c1b6a05b096a7feb5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38713 } }
I20260927 04:02:32.982774 4916 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1da49ee0527f49c1b6a05b096a7feb5e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:02:32.982815 4916 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1da49ee0527f49c1b6a05b096a7feb5e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:02:32.982861 4916 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1da49ee0527f49c1b6a05b096a7feb5e [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:02:32.983656 4916 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1da49ee0527f49c1b6a05b096a7feb5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1da49ee0527f49c1b6a05b096a7feb5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38713 } }
I20260927 04:02:32.983769 4916 leader_election.cc:302] T 00000000000000000000000000000000 P 1da49ee0527f49c1b6a05b096a7feb5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1da49ee0527f49c1b6a05b096a7feb5e; no voters:
I20260927 04:02:32.983915 4916 leader_election.cc:288] T 00000000000000000000000000000000 P 1da49ee0527f49c1b6a05b096a7feb5e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:02:32.983984 4921 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1da49ee0527f49c1b6a05b096a7feb5e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:02:32.984156 4916 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1da49ee0527f49c1b6a05b096a7feb5e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:02:32.984143 4921 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1da49ee0527f49c1b6a05b096a7feb5e [term 1 LEADER]: Becoming Leader. State: Replica: 1da49ee0527f49c1b6a05b096a7feb5e, State: Running, Role: LEADER
I20260927 04:02:32.984306 4921 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1da49ee0527f49c1b6a05b096a7feb5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1da49ee0527f49c1b6a05b096a7feb5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38713 } }
I20260927 04:02:32.984787 4923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1da49ee0527f49c1b6a05b096a7feb5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1da49ee0527f49c1b6a05b096a7feb5e. Latest consensus state: current_term: 1 leader_uuid: "1da49ee0527f49c1b6a05b096a7feb5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1da49ee0527f49c1b6a05b096a7feb5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38713 } } }
I20260927 04:02:32.984856 4923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1da49ee0527f49c1b6a05b096a7feb5e [sys.catalog]: This master's current role is: LEADER
I20260927 04:02:32.984813 4922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1da49ee0527f49c1b6a05b096a7feb5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1da49ee0527f49c1b6a05b096a7feb5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1da49ee0527f49c1b6a05b096a7feb5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38713 } } }
I20260927 04:02:32.984897 4922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1da49ee0527f49c1b6a05b096a7feb5e [sys.catalog]: This master's current role is: LEADER
I20260927 04:02:32.985038 4930 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:02:32.985531 4930 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:02:32.987913 4930 catalog_manager.cc:1347] Generated new cluster ID: 0ff5b7501b164959af93e6b19188b940
I20260927 04:02:32.987972 4930 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 04:02:32 dist-test-slave-glgz krb5kdc[4835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 04:02:32.992363 4920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:32.980567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.151:36631 (local address 127.0.0.1:38713)
0927 04:02:32.980781 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:32.980792 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:32.980827 (+ 35us) server_negotiation.cc:407] Connection header received
0927 04:02:32.981479 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:32.981489 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:32.981558 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:32.981722 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:32.982328 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:32.984076 (+ 1748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:32.985227 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:32.985970 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:32.986093 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:32.988957 (+ 2864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:32.988961 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:32.988961 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:32.988974 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:32.990668 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:32.991105 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:32.991115 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:32.991116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:32.991226 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:32.991471 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:32.991472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:32.991472 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:32.991717 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:32.991931 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:32.992073 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 04:02:32.992210 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":120,"threads_started":1}
I20260927 04:02:32.993409 16101 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 04:02:33.001216 4920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:32.994860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.151:33555 (local address 127.0.0.1:38713)
0927 04:02:32.994903 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:32.994905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:32.995025 (+ 120us) server_negotiation.cc:407] Connection header received
0927 04:02:32.995156 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:32.995156 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:32.995167 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:32.995211 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:32.995731 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:32.996256 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:32.996982 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:32.997221 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:32.997299 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:32.998516 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:32.998518 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:32.998518 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:32.998525 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:33.000033 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:33.000401 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:33.000402 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:33.000402 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:33.000457 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:33.000691 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:33.000692 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:33.000693 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:33.000807 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:33.000931 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:33.001052 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 04:02:33.001113 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 04:02:33.008212 4930 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:02:33.008826 4930 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:02:33.013377 4930 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1da49ee0527f49c1b6a05b096a7feb5e: Generated new TSK 0
I20260927 04:02:33.013607 4930 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 04:02:33.016382 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 4848
W20260927 04:02:33.020681 4833 connection.cc:535] client connection to 127.0.0.1:38713 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38713): Connection reset by peer (error 104)
2026-09-27T04:02:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1401 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481498018407-16101-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:02:33 dist-test-slave-glgz krb5kdc[4947](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:02:33 dist-test-slave-glgz krb5kdc[4947](info): set up 2 sockets
Sep 27 04:02:33 dist-test-slave-glgz krb5kdc[4947](info): commencing operation
krb5kdc: starting...
W20260927 04:02:34.104020 16101 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Sep 27 04:02:34 dist-test-slave-glgz krb5kdc[4947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:02:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:02:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481498018407-16101-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481498018407-16101-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 04:02:34.226096 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481498018407-16101-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481498018407-16101-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481498018407-16101-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44189
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481498018407-16101-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44189
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:02:34.306977 4960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:02:34.307224 4960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:02:34.307248 4960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:02:34.308531 4960 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:02:34.308583 4960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:02:34.308598 4960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:02:34.308610 4960 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:02:34.308681 4960 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:02:34.310298 4960 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481498018407-16101-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481498018407-16101-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44189
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481498018407-16101-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481498018407-16101-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:02:34.310560 4960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:02:34.310811 4960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:02:34.315706 4965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:34.315745 4960 server_base.cc:1034] running on GCE node
W20260927 04:02:34.315785 4966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:34.315713 4968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:34.316066 4960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:02:34.316383 4960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:02:34.317526 4960 hybrid_clock.cc:648] HybridClock initialized: now 1790481754317507 us; error 39 us; skew 500 ppm
Sep 27 04:02:34 dist-test-slave-glgz krb5kdc[4947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481754, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:02:34.319986 4960 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 04:02:34.320499 4960 webserver.cc:460] Webserver started at http://127.0.0.1:38559/ using document root <none> and password file <none>
I20260927 04:02:34.320746 4960 fs_manager.cc:359] Metadata directory not provided
I20260927 04:02:34.320813 4960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:02:34.320971 4960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:02:34.322062 4960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481498018407-16101-0/minicluster-data/master-0/data/instance:
uuid: "16be2bb0a07c4671b22cab7780d95686"
format_stamp: "Formatted at 2026-09-27 04:02:34 on dist-test-slave-glgz"
server_key: "46018e233f2fb4a63227b8c79f3d4889"
server_key_iv: "699e3b9275aaaba920fe2c9eb61697a3"
server_key_version: "encryptionkey@0"
I20260927 04:02:34.322425 4960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481498018407-16101-0/minicluster-data/master-0/wal/instance:
uuid: "16be2bb0a07c4671b22cab7780d95686"
format_stamp: "Formatted at 2026-09-27 04:02:34 on dist-test-slave-glgz"
server_key: "46018e233f2fb4a63227b8c79f3d4889"
server_key_iv: "699e3b9275aaaba920fe2c9eb61697a3"
server_key_version: "encryptionkey@0"
I20260927 04:02:34.323899 4960 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 04:02:34.324683 4975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:34.324903 4960 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:02:34.325008 4960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481498018407-16101-0/minicluster-data/master-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481498018407-16101-0/minicluster-data/master-0/wal
uuid: "16be2bb0a07c4671b22cab7780d95686"
format_stamp: "Formatted at 2026-09-27 04:02:34 on dist-test-slave-glgz"
server_key: "46018e233f2fb4a63227b8c79f3d4889"
server_key_iv: "699e3b9275aaaba920fe2c9eb61697a3"
server_key_version: "encryptionkey@0"
I20260927 04:02:34.325091 4960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481498018407-16101-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481498018407-16101-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481498018407-16101-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:02:34.342108 4960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:02:34.344084 4960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:02:34.344262 4960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:02:34.348351 4960 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44189
I20260927 04:02:34.348389 5027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44189 every 8 connection(s)
I20260927 04:02:34.348776 4960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
I20260927 04:02:34.349644 5028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:02:34.350847 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 4960
I20260927 04:02:34.350993 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481498018407-16101-0/minicluster-data/master-0/wal/instance
I20260927 04:02:34.351225 16101 external_mini_cluster.cc:1428] Setting key 6c2ba40915059e8c180d92edb51762a3
I20260927 04:02:34.352514 5028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16be2bb0a07c4671b22cab7780d95686: Bootstrap starting.
I20260927 04:02:34.353232 5028 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16be2bb0a07c4671b22cab7780d95686: Neither blocks nor log segments found. Creating new log.
I20260927 04:02:34.353535 5028 log.cc:824] T 00000000000000000000000000000000 P 16be2bb0a07c4671b22cab7780d95686: Log is configured to *not* fsync() on all Append() calls
I20260927 04:02:34.354285 5028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16be2bb0a07c4671b22cab7780d95686: No bootstrap required, opened a new log
I20260927 04:02:34.356707 5028 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16be2bb0a07c4671b22cab7780d95686 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16be2bb0a07c4671b22cab7780d95686" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44189 } }
I20260927 04:02:34.356848 5028 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16be2bb0a07c4671b22cab7780d95686 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:02:34.356869 5028 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16be2bb0a07c4671b22cab7780d95686 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16be2bb0a07c4671b22cab7780d95686, State: Initialized, Role: FOLLOWER
I20260927 04:02:34.357000 5028 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16be2bb0a07c4671b22cab7780d95686 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16be2bb0a07c4671b22cab7780d95686" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44189 } }
I20260927 04:02:34.357081 5028 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16be2bb0a07c4671b22cab7780d95686 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:02:34.357142 5028 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16be2bb0a07c4671b22cab7780d95686 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:02:34.357199 5028 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16be2bb0a07c4671b22cab7780d95686 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:02:34.357523 5031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:34.351832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.151:33795 (local address 127.0.0.1:44189)
0927 04:02:34.352019 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:34.352028 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:34.352057 (+ 29us) server_negotiation.cc:407] Connection header received
0927 04:02:34.352469 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:34.352475 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:34.352535 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:34.352697 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:34.352931 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:34.354144 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:34.355298 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:34.355787 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:34.355877 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:34.356649 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:34.356653 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:34.356653 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:02:34.356820 (+ 167us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:34.356878 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:02:34.356894 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:34.357000 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:34.357285 (+ 285us) server_negotiation.cc:299] Negotiation successful
0927 04:02:34.357318 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":85,"threads_started":1}
I20260927 04:02:34.357924 5028 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16be2bb0a07c4671b22cab7780d95686 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16be2bb0a07c4671b22cab7780d95686" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44189 } }
I20260927 04:02:34.358047 5028 leader_election.cc:302] T 00000000000000000000000000000000 P 16be2bb0a07c4671b22cab7780d95686 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16be2bb0a07c4671b22cab7780d95686; no voters:
I20260927 04:02:34.358218 5028 leader_election.cc:288] T 00000000000000000000000000000000 P 16be2bb0a07c4671b22cab7780d95686 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:02:34.358246 5033 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16be2bb0a07c4671b22cab7780d95686 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:02:34.358404 5033 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16be2bb0a07c4671b22cab7780d95686 [term 1 LEADER]: Becoming Leader. State: Replica: 16be2bb0a07c4671b22cab7780d95686, State: Running, Role: LEADER
I20260927 04:02:34.358489 5028 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16be2bb0a07c4671b22cab7780d95686 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:02:34.358490 5033 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16be2bb0a07c4671b22cab7780d95686 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16be2bb0a07c4671b22cab7780d95686" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44189 } }
I20260927 04:02:34.358579 16101 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 04:02:34.358912 5033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16be2bb0a07c4671b22cab7780d95686 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16be2bb0a07c4671b22cab7780d95686. Latest consensus state: current_term: 1 leader_uuid: "16be2bb0a07c4671b22cab7780d95686" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16be2bb0a07c4671b22cab7780d95686" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44189 } } }
I20260927 04:02:34.358984 5033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16be2bb0a07c4671b22cab7780d95686 [sys.catalog]: This master's current role is: LEADER
I20260927 04:02:34.359258 5040 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:02:34.359997 5040 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:02:34.360126 5034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16be2bb0a07c4671b22cab7780d95686 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16be2bb0a07c4671b22cab7780d95686" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16be2bb0a07c4671b22cab7780d95686" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44189 } } }
I20260927 04:02:34.360415 5034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16be2bb0a07c4671b22cab7780d95686 [sys.catalog]: This master's current role is: LEADER
I20260927 04:02:34.362036 5040 catalog_manager.cc:1347] Generated new cluster ID: 3faa0eb77a4a41b28cb7ff43c18175c9
I20260927 04:02:34.362089 5040 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:02:34.364339 5031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:34.360342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.151:41475 (local address 127.0.0.1:44189)
0927 04:02:34.360377 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:34.360379 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:34.360645 (+ 266us) server_negotiation.cc:407] Connection header received
0927 04:02:34.360695 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:34.360696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:34.360708 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:34.360768 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:34.361224 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:34.361892 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:34.362919 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:34.363248 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:34.363301 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:34.364062 (+ 761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:34.364063 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:34.364064 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:02:34.364067 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:34.364080 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:02:34.364084 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:34.364124 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:34.364259 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 04:02:34.364268 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 04:02:34.367210 5040 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:02:34.367720 5040 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:02:34.373814 5040 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16be2bb0a07c4671b22cab7780d95686: Generated new TSK 0
I20260927 04:02:34.373999 5040 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 04:02:34.380260 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 4960
W20260927 04:02:34.384383 4945 connection.cc:535] client connection to 127.0.0.1:44189 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44189): Connection reset by peer (error 104)
2026-09-27T04:02:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1363 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481498018407-16101-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:02:34 dist-test-slave-glgz krb5kdc[5059](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:02:34 dist-test-slave-glgz krb5kdc[5059](info): set up 2 sockets
Sep 27 04:02:34 dist-test-slave-glgz krb5kdc[5059](info): commencing operation
krb5kdc: starting...
W20260927 04:02:35.427673 16101 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Sep 27 04:02:35 dist-test-slave-glgz krb5kdc[5059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:02:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:02:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481498018407-16101-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481498018407-16101-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 04:02:35.545770 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481498018407-16101-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481498018407-16101-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481498018407-16101-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43469
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481498018407-16101-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43469
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:02:35.626124 5072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:02:35.626387 5072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:02:35.626427 5072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:02:35.627593 5072 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:02:35.627671 5072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:02:35.627694 5072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:02:35.627722 5072 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:02:35.627763 5072 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:02:35.629261 5072 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481498018407-16101-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481498018407-16101-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43469
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481498018407-16101-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481498018407-16101-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:02:35.629547 5072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:02:35.629776 5072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:02:35.634867 5077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:35.634867 5080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:35.634862 5078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:35.635245 5072 server_base.cc:1034] running on GCE node
I20260927 04:02:35.635499 5072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:02:35.635785 5072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:02:35.636961 5072 hybrid_clock.cc:648] HybridClock initialized: now 1790481755636948 us; error 27 us; skew 500 ppm
Sep 27 04:02:35 dist-test-slave-glgz krb5kdc[5059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481755, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:02:35.639739 5072 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 04:02:35.640219 5072 webserver.cc:460] Webserver started at http://127.0.0.1:41707/ using document root <none> and password file <none>
I20260927 04:02:35.640455 5072 fs_manager.cc:359] Metadata directory not provided
I20260927 04:02:35.640501 5072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:02:35.640677 5072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:02:35.641963 5072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481498018407-16101-0/minicluster-data/master-0/data/instance:
uuid: "d64b56f8f266440f88e35f8b6b154295"
format_stamp: "Formatted at 2026-09-27 04:02:35 on dist-test-slave-glgz"
server_key: "85fc60d6b7b2987309b46f3654a8ea64"
server_key_iv: "10acc187335008f76fcc8c5d77728e9b"
server_key_version: "encryptionkey@0"
I20260927 04:02:35.642359 5072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481498018407-16101-0/minicluster-data/master-0/wal/instance:
uuid: "d64b56f8f266440f88e35f8b6b154295"
format_stamp: "Formatted at 2026-09-27 04:02:35 on dist-test-slave-glgz"
server_key: "85fc60d6b7b2987309b46f3654a8ea64"
server_key_iv: "10acc187335008f76fcc8c5d77728e9b"
server_key_version: "encryptionkey@0"
I20260927 04:02:35.643731 5072 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 04:02:35.644456 5087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:35.644677 5072 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:02:35.644770 5072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481498018407-16101-0/minicluster-data/master-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481498018407-16101-0/minicluster-data/master-0/wal
uuid: "d64b56f8f266440f88e35f8b6b154295"
format_stamp: "Formatted at 2026-09-27 04:02:35 on dist-test-slave-glgz"
server_key: "85fc60d6b7b2987309b46f3654a8ea64"
server_key_iv: "10acc187335008f76fcc8c5d77728e9b"
server_key_version: "encryptionkey@0"
I20260927 04:02:35.644855 5072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481498018407-16101-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481498018407-16101-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481498018407-16101-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:02:35.648240 5072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:02:35.650102 5072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:02:35.650254 5072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:02:35.654192 5072 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43469
I20260927 04:02:35.654320 5139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43469 every 8 connection(s)
I20260927 04:02:35.654598 5072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
I20260927 04:02:35.655588 5140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:02:35.658099 5140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d64b56f8f266440f88e35f8b6b154295: Bootstrap starting.
I20260927 04:02:35.658901 5140 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d64b56f8f266440f88e35f8b6b154295: Neither blocks nor log segments found. Creating new log.
I20260927 04:02:35.659260 5140 log.cc:824] T 00000000000000000000000000000000 P d64b56f8f266440f88e35f8b6b154295: Log is configured to *not* fsync() on all Append() calls
I20260927 04:02:35.660028 5140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d64b56f8f266440f88e35f8b6b154295: No bootstrap required, opened a new log
I20260927 04:02:35.660353 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 5072
I20260927 04:02:35.660502 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481498018407-16101-0/minicluster-data/master-0/wal/instance
I20260927 04:02:35.660741 16101 external_mini_cluster.cc:1428] Setting key afd64afc9d98b259239e451c7e82c04e
I20260927 04:02:35.662366 5140 raft_consensus.cc:348] T 00000000000000000000000000000000 P d64b56f8f266440f88e35f8b6b154295 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d64b56f8f266440f88e35f8b6b154295" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43469 } }
I20260927 04:02:35.662581 5140 raft_consensus.cc:374] T 00000000000000000000000000000000 P d64b56f8f266440f88e35f8b6b154295 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:02:35.662616 5140 raft_consensus.cc:729] T 00000000000000000000000000000000 P d64b56f8f266440f88e35f8b6b154295 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d64b56f8f266440f88e35f8b6b154295, State: Initialized, Role: FOLLOWER
I20260927 04:02:35.662755 5140 consensus_queue.cc:260] T 00000000000000000000000000000000 P d64b56f8f266440f88e35f8b6b154295 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d64b56f8f266440f88e35f8b6b154295" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43469 } }
I20260927 04:02:35.662842 5140 raft_consensus.cc:388] T 00000000000000000000000000000000 P d64b56f8f266440f88e35f8b6b154295 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:02:35.662881 5140 raft_consensus.cc:482] T 00000000000000000000000000000000 P d64b56f8f266440f88e35f8b6b154295 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:02:35.662940 5140 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d64b56f8f266440f88e35f8b6b154295 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:02:35.663563 5140 raft_consensus.cc:504] T 00000000000000000000000000000000 P d64b56f8f266440f88e35f8b6b154295 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d64b56f8f266440f88e35f8b6b154295" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43469 } }
I20260927 04:02:35.663686 5140 leader_election.cc:302] T 00000000000000000000000000000000 P d64b56f8f266440f88e35f8b6b154295 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d64b56f8f266440f88e35f8b6b154295; no voters:
I20260927 04:02:35.663877 5140 leader_election.cc:288] T 00000000000000000000000000000000 P d64b56f8f266440f88e35f8b6b154295 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:02:35.664018 5145 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d64b56f8f266440f88e35f8b6b154295 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:02:35.664116 5144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:35.661281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.151:38965 (local address 127.0.0.1:43469)
0927 04:02:35.661512 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:35.661523 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:35.661556 (+ 33us) server_negotiation.cc:407] Connection header received
0927 04:02:35.662572 (+ 1016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:35.662582 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:35.662621 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:35.662778 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:35.663560 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:35.663565 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:35.663565 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:02:35.663569 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:35.663620 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:02:35.663636 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:35.663713 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:35.663847 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 04:02:35.663897 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":108,"threads_started":1}
I20260927 04:02:35.664206 5145 raft_consensus.cc:686] T 00000000000000000000000000000000 P d64b56f8f266440f88e35f8b6b154295 [term 1 LEADER]: Becoming Leader. State: Replica: d64b56f8f266440f88e35f8b6b154295, State: Running, Role: LEADER
I20260927 04:02:35.664166 5140 sys_catalog.cc:564] T 00000000000000000000000000000000 P d64b56f8f266440f88e35f8b6b154295 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:02:35.664307 5145 consensus_queue.cc:237] T 00000000000000000000000000000000 P d64b56f8f266440f88e35f8b6b154295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d64b56f8f266440f88e35f8b6b154295" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43469 } }
I20260927 04:02:35.664939 5146 sys_catalog.cc:455] T 00000000000000000000000000000000 P d64b56f8f266440f88e35f8b6b154295 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d64b56f8f266440f88e35f8b6b154295" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d64b56f8f266440f88e35f8b6b154295" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43469 } } }
I20260927 04:02:35.665150 16101 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 04:02:35.665180 5146 sys_catalog.cc:458] T 00000000000000000000000000000000 P d64b56f8f266440f88e35f8b6b154295 [sys.catalog]: This master's current role is: LEADER
I20260927 04:02:35.665107 5147 sys_catalog.cc:455] T 00000000000000000000000000000000 P d64b56f8f266440f88e35f8b6b154295 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d64b56f8f266440f88e35f8b6b154295. Latest consensus state: current_term: 1 leader_uuid: "d64b56f8f266440f88e35f8b6b154295" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d64b56f8f266440f88e35f8b6b154295" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43469 } } }
I20260927 04:02:35.665330 5147 sys_catalog.cc:458] T 00000000000000000000000000000000 P d64b56f8f266440f88e35f8b6b154295 [sys.catalog]: This master's current role is: LEADER
W20260927 04:02:35.666229 5161 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d64b56f8f266440f88e35f8b6b154295: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 04:02:35.666595 5161 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 04:02:35.666755 5154 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:02:35.666934 5154 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:02:35.668550 5144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:35.666723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.151:57083 (local address 127.0.0.1:43469)
0927 04:02:35.667606 (+ 883us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:35.667608 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:35.667617 (+ 9us) server_negotiation.cc:407] Connection header received
0927 04:02:35.667637 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:35.667638 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:35.667642 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:35.667705 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:35.668311 (+ 606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:35.668313 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:35.668313 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:02:35.668317 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:35.668331 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:02:35.668336 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:35.668365 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:35.668474 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 04:02:35.668488 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":865}
I20260927 04:02:35.668850 5154 catalog_manager.cc:1347] Generated new cluster ID: da33814c35174727aab5890eb08f08cf
I20260927 04:02:35.668900 5154 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:02:35.688158 5154 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:02:35.688364 5154 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:02:35.693290 5154 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d64b56f8f266440f88e35f8b6b154295: Generated new TSK 0
I20260927 04:02:35.693533 5154 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 04:02:35.695827 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 5072
2026-09-27T04:02:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1315 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481498018407-16101-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:02:35 dist-test-slave-glgz krb5kdc[5171](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:02:35 dist-test-slave-glgz krb5kdc[5171](info): set up 2 sockets
Sep 27 04:02:35 dist-test-slave-glgz krb5kdc[5171](info): commencing operation
krb5kdc: starting...
W20260927 04:02:36.743611 16101 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Sep 27 04:02:36 dist-test-slave-glgz krb5kdc[5171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:02:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:02:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.185.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab.
Entry for principal kudu/127.15.185.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab.
I20260927 04:02:36.865751 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481498018407-16101-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481498018407-16101-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481498018407-16101-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:40787
--webserver_interface=127.15.185.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:45221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.185.126:40787
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:02:36.947994 5184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:02:36.948259 5184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:02:36.948298 5184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:02:36.949493 5184 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:02:36.949574 5184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:02:36.949608 5184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:02:36.949642 5184 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:02:36.949672 5184 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:02:36.951212 5184 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:45221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481498018407-16101-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481498018407-16101-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.185.126:40787
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:40787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481498018407-16101-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:02:36.951503 5184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:02:36.951742 5184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:02:36.956771 5189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:36.956827 5190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:36.956972 5184 server_base.cc:1034] running on GCE node
W20260927 04:02:36.956985 5192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:36.957260 5184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:02:36.957546 5184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:02:36.958693 5184 hybrid_clock.cc:648] HybridClock initialized: now 1790481756958675 us; error 46 us; skew 500 ppm
Sep 27 04:02:36 dist-test-slave-glgz krb5kdc[5171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481756, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:02:36.966259 5184 init.cc:377] Logged in from keytab as kudu/127.15.185.126@KRBTEST.COM (short username kudu)
I20260927 04:02:36.966676 5184 webserver.cc:460] Webserver started at http://127.15.185.126:35585/ using document root <none> and password file <none>
I20260927 04:02:36.966881 5184 fs_manager.cc:359] Metadata directory not provided
I20260927 04:02:36.966920 5184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:02:36.967011 5184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:02:36.968156 5184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481498018407-16101-0/minicluster-data/master-0/data/instance:
uuid: "aba441680ca3462087198d4707dfd375"
format_stamp: "Formatted at 2026-09-27 04:02:36 on dist-test-slave-glgz"
server_key: "2dd463715d8184fe20dab5b87a1e2b1b"
server_key_iv: "b819a5d702eb0ccf2a64465796e02792"
server_key_version: "encryptionkey@0"
I20260927 04:02:36.968484 5184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481498018407-16101-0/minicluster-data/master-0/wal/instance:
uuid: "aba441680ca3462087198d4707dfd375"
format_stamp: "Formatted at 2026-09-27 04:02:36 on dist-test-slave-glgz"
server_key: "2dd463715d8184fe20dab5b87a1e2b1b"
server_key_iv: "b819a5d702eb0ccf2a64465796e02792"
server_key_version: "encryptionkey@0"
I20260927 04:02:36.969750 5184 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 04:02:36.970475 5199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:36.970628 5184 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 04:02:36.970709 5184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481498018407-16101-0/minicluster-data/master-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481498018407-16101-0/minicluster-data/master-0/wal
uuid: "aba441680ca3462087198d4707dfd375"
format_stamp: "Formatted at 2026-09-27 04:02:36 on dist-test-slave-glgz"
server_key: "2dd463715d8184fe20dab5b87a1e2b1b"
server_key_iv: "b819a5d702eb0ccf2a64465796e02792"
server_key_version: "encryptionkey@0"
I20260927 04:02:36.970780 5184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481498018407-16101-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481498018407-16101-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481498018407-16101-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:02:36.989218 5184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:02:36.991262 5184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:02:36.991379 5184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:02:36.995007 5184 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.126:40787
I20260927 04:02:36.995129 5251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.126:40787 every 8 connection(s)
I20260927 04:02:36.995486 5184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
I20260927 04:02:36.996462 5252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:02:36.998732 5252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aba441680ca3462087198d4707dfd375: Bootstrap starting.
I20260927 04:02:36.999310 5252 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aba441680ca3462087198d4707dfd375: Neither blocks nor log segments found. Creating new log.
I20260927 04:02:36.999549 5252 log.cc:824] T 00000000000000000000000000000000 P aba441680ca3462087198d4707dfd375: Log is configured to *not* fsync() on all Append() calls
I20260927 04:02:37.000209 5252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aba441680ca3462087198d4707dfd375: No bootstrap required, opened a new log
I20260927 04:02:37.000694 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 5184
I20260927 04:02:37.000794 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481498018407-16101-0/minicluster-data/master-0/wal/instance
I20260927 04:02:37.000977 16101 external_mini_cluster.cc:1428] Setting key 07fe495b77abaed40af09f9250340131
I20260927 04:02:37.002350 5252 raft_consensus.cc:348] T 00000000000000000000000000000000 P aba441680ca3462087198d4707dfd375 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aba441680ca3462087198d4707dfd375" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 40787 } }
I20260927 04:02:37.002538 5252 raft_consensus.cc:374] T 00000000000000000000000000000000 P aba441680ca3462087198d4707dfd375 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:02:37.002607 5252 raft_consensus.cc:729] T 00000000000000000000000000000000 P aba441680ca3462087198d4707dfd375 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aba441680ca3462087198d4707dfd375, State: Initialized, Role: FOLLOWER
I20260927 04:02:37.002756 5252 consensus_queue.cc:260] T 00000000000000000000000000000000 P aba441680ca3462087198d4707dfd375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aba441680ca3462087198d4707dfd375" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 40787 } }
I20260927 04:02:37.002844 5252 raft_consensus.cc:388] T 00000000000000000000000000000000 P aba441680ca3462087198d4707dfd375 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:02:37.002882 5252 raft_consensus.cc:482] T 00000000000000000000000000000000 P aba441680ca3462087198d4707dfd375 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:02:37.002933 5252 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aba441680ca3462087198d4707dfd375 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:02:37.004228 5252 raft_consensus.cc:504] T 00000000000000000000000000000000 P aba441680ca3462087198d4707dfd375 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aba441680ca3462087198d4707dfd375" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 40787 } }
I20260927 04:02:37.004331 5252 leader_election.cc:302] T 00000000000000000000000000000000 P aba441680ca3462087198d4707dfd375 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aba441680ca3462087198d4707dfd375; no voters:
I20260927 04:02:37.004590 5252 leader_election.cc:288] T 00000000000000000000000000000000 P aba441680ca3462087198d4707dfd375 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:02:37.004612 5257 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aba441680ca3462087198d4707dfd375 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:02:37.004734 5257 raft_consensus.cc:686] T 00000000000000000000000000000000 P aba441680ca3462087198d4707dfd375 [term 1 LEADER]: Becoming Leader. State: Replica: aba441680ca3462087198d4707dfd375, State: Running, Role: LEADER
I20260927 04:02:37.004885 5252 sys_catalog.cc:564] T 00000000000000000000000000000000 P aba441680ca3462087198d4707dfd375 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:02:37.004885 5257 consensus_queue.cc:237] T 00000000000000000000000000000000 P aba441680ca3462087198d4707dfd375 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aba441680ca3462087198d4707dfd375" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 40787 } }
I20260927 04:02:37.005455 5259 sys_catalog.cc:455] T 00000000000000000000000000000000 P aba441680ca3462087198d4707dfd375 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aba441680ca3462087198d4707dfd375. Latest consensus state: current_term: 1 leader_uuid: "aba441680ca3462087198d4707dfd375" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aba441680ca3462087198d4707dfd375" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 40787 } } }
I20260927 04:02:37.005467 5258 sys_catalog.cc:455] T 00000000000000000000000000000000 P aba441680ca3462087198d4707dfd375 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aba441680ca3462087198d4707dfd375" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aba441680ca3462087198d4707dfd375" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 40787 } } }
I20260927 04:02:37.005657 5259 sys_catalog.cc:458] T 00000000000000000000000000000000 P aba441680ca3462087198d4707dfd375 [sys.catalog]: This master's current role is: LEADER
I20260927 04:02:37.005671 5258 sys_catalog.cc:458] T 00000000000000000000000000000000 P aba441680ca3462087198d4707dfd375 [sys.catalog]: This master's current role is: LEADER
I20260927 04:02:37.005955 5266 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:02:37.006651 5266 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 04:02:37 dist-test-slave-glgz krb5kdc[5171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 04:02:37.010067 5266 catalog_manager.cc:1347] Generated new cluster ID: b4db455717af4455b5287a173809f7d8
I20260927 04:02:37.010131 5266 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:02:37.012940 5256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:37.001615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40353 (local address 127.15.185.126:40787)
0927 04:02:37.001796 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:37.001804 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:37.001829 (+ 25us) server_negotiation.cc:407] Connection header received
0927 04:02:37.002344 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:37.002352 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:37.002407 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:37.002529 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:37.003609 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:37.004628 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:37.005623 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:37.006237 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:37.006675 (+ 438us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:37.009376 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:37.009380 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:37.009380 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:37.009390 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:37.011376 (+ 1986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:37.011786 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:37.011789 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:37.011790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:37.011904 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:37.012150 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:37.012152 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:37.012152 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:37.012392 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:37.012579 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:37.012666 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 04:02:37.012767 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":86,"threads_started":1}
I20260927 04:02:37.014039 16101 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 04:02:37.025979 5256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:37.019606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36213 (local address 127.15.185.126:40787)
0927 04:02:37.019683 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:37.019685 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:37.019846 (+ 161us) server_negotiation.cc:407] Connection header received
0927 04:02:37.019924 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:37.019925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:37.019934 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:37.019978 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:37.020516 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:37.021009 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:37.021763 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:37.021992 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:37.022066 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:37.023357 (+ 1291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:37.023358 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:37.023359 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:37.023366 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:37.024787 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:37.025152 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:37.025153 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:37.025154 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:37.025218 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:37.025445 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:37.025446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:37.025446 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:37.025555 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:37.025683 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:37.025799 (+ 116us) server_negotiation.cc:299] Negotiation successful
0927 04:02:37.025858 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":12,"server-negotiator.queue_time_us":38}
I20260927 04:02:37.031265 5266 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:02:37.031925 5266 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:02:37.034797 5266 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aba441680ca3462087198d4707dfd375: Generated new TSK 0
I20260927 04:02:37.034998 5266 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 04:02:37.037881 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 5184
W20260927 04:02:37.042029 5169 connection.cc:535] client connection to 127.15.185.126:40787 recv error: Network error: failed to read from TLS socket (remote: 127.15.185.126:40787): Connection reset by peer (error 104)
2026-09-27T04:02:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1342 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481498018407-16101-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:02:37 dist-test-slave-glgz krb5kdc[5283](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:02:37 dist-test-slave-glgz krb5kdc[5283](info): set up 2 sockets
Sep 27 04:02:37 dist-test-slave-glgz krb5kdc[5283](info): commencing operation
krb5kdc: starting...
W20260927 04:02:38.079797 16101 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Sep 27 04:02:38 dist-test-slave-glgz krb5kdc[5283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:02:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:02:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.185.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab.
Entry for principal kudu/127.15.185.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab.
I20260927 04:02:38.197911 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481498018407-16101-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481498018407-16101-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481498018407-16101-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:35045
--webserver_interface=127.15.185.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:38425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.185.126:35045
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:02:38.281867 5296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:02:38.282110 5296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:02:38.282161 5296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:02:38.283715 5296 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:02:38.283775 5296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:02:38.283792 5296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:02:38.283806 5296 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:02:38.283874 5296 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:02:38.285497 5296 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:38425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481498018407-16101-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481498018407-16101-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.185.126:35045
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:35045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481498018407-16101-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:02:38.285784 5296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:02:38.286027 5296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:02:38.291000 5302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:38.290971 5301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:38.290982 5304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:38.291232 5296 server_base.cc:1034] running on GCE node
I20260927 04:02:38.291539 5296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:02:38.291800 5296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:02:38.292974 5296 hybrid_clock.cc:648] HybridClock initialized: now 1790481758292947 us; error 54 us; skew 500 ppm
Sep 27 04:02:38 dist-test-slave-glgz krb5kdc[5283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481758, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:02:38.295567 5296 init.cc:377] Logged in from keytab as kudu/127.15.185.126@KRBTEST.COM (short username kudu)
I20260927 04:02:38.295971 5296 webserver.cc:460] Webserver started at http://127.15.185.126:33533/ using document root <none> and password file <none>
I20260927 04:02:38.296176 5296 fs_manager.cc:359] Metadata directory not provided
I20260927 04:02:38.296217 5296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:02:38.296304 5296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:02:38.297326 5296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481498018407-16101-0/minicluster-data/master-0/data/instance:
uuid: "822e9a368ef647bb8ba01bf4fd14ca26"
format_stamp: "Formatted at 2026-09-27 04:02:38 on dist-test-slave-glgz"
server_key: "61daa3c38fe1302958ca680f5ca16927"
server_key_iv: "d0980028bebdb391212a49a01e70df2b"
server_key_version: "encryptionkey@0"
I20260927 04:02:38.297623 5296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481498018407-16101-0/minicluster-data/master-0/wal/instance:
uuid: "822e9a368ef647bb8ba01bf4fd14ca26"
format_stamp: "Formatted at 2026-09-27 04:02:38 on dist-test-slave-glgz"
server_key: "61daa3c38fe1302958ca680f5ca16927"
server_key_iv: "d0980028bebdb391212a49a01e70df2b"
server_key_version: "encryptionkey@0"
I20260927 04:02:38.298943 5296 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 04:02:38.299700 5311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:38.299876 5296 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 04:02:38.299966 5296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481498018407-16101-0/minicluster-data/master-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481498018407-16101-0/minicluster-data/master-0/wal
uuid: "822e9a368ef647bb8ba01bf4fd14ca26"
format_stamp: "Formatted at 2026-09-27 04:02:38 on dist-test-slave-glgz"
server_key: "61daa3c38fe1302958ca680f5ca16927"
server_key_iv: "d0980028bebdb391212a49a01e70df2b"
server_key_version: "encryptionkey@0"
I20260927 04:02:38.300041 5296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481498018407-16101-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481498018407-16101-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481498018407-16101-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:02:38.316280 5296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:02:38.318768 5296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:02:38.318934 5296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:02:38.322664 5296 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.126:35045
I20260927 04:02:38.322734 5363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.126:35045 every 8 connection(s)
I20260927 04:02:38.323101 5296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
I20260927 04:02:38.324013 5364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:02:38.326331 5364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 822e9a368ef647bb8ba01bf4fd14ca26: Bootstrap starting.
I20260927 04:02:38.326922 5364 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 822e9a368ef647bb8ba01bf4fd14ca26: Neither blocks nor log segments found. Creating new log.
I20260927 04:02:38.327303 5364 log.cc:824] T 00000000000000000000000000000000 P 822e9a368ef647bb8ba01bf4fd14ca26: Log is configured to *not* fsync() on all Append() calls
I20260927 04:02:38.328006 5364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 822e9a368ef647bb8ba01bf4fd14ca26: No bootstrap required, opened a new log
I20260927 04:02:38.329597 5364 raft_consensus.cc:348] T 00000000000000000000000000000000 P 822e9a368ef647bb8ba01bf4fd14ca26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "822e9a368ef647bb8ba01bf4fd14ca26" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 35045 } }
I20260927 04:02:38.329761 5364 raft_consensus.cc:374] T 00000000000000000000000000000000 P 822e9a368ef647bb8ba01bf4fd14ca26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:02:38.329806 5364 raft_consensus.cc:729] T 00000000000000000000000000000000 P 822e9a368ef647bb8ba01bf4fd14ca26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 822e9a368ef647bb8ba01bf4fd14ca26, State: Initialized, Role: FOLLOWER
I20260927 04:02:38.329974 5364 consensus_queue.cc:260] T 00000000000000000000000000000000 P 822e9a368ef647bb8ba01bf4fd14ca26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "822e9a368ef647bb8ba01bf4fd14ca26" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 35045 } }
I20260927 04:02:38.330057 5364 raft_consensus.cc:388] T 00000000000000000000000000000000 P 822e9a368ef647bb8ba01bf4fd14ca26 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:02:38.330096 5364 raft_consensus.cc:482] T 00000000000000000000000000000000 P 822e9a368ef647bb8ba01bf4fd14ca26 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:02:38.330158 5364 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 822e9a368ef647bb8ba01bf4fd14ca26 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:02:38.331064 5364 raft_consensus.cc:504] T 00000000000000000000000000000000 P 822e9a368ef647bb8ba01bf4fd14ca26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "822e9a368ef647bb8ba01bf4fd14ca26" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 35045 } }
I20260927 04:02:38.331193 5364 leader_election.cc:302] T 00000000000000000000000000000000 P 822e9a368ef647bb8ba01bf4fd14ca26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 822e9a368ef647bb8ba01bf4fd14ca26; no voters:
I20260927 04:02:38.331372 5364 leader_election.cc:288] T 00000000000000000000000000000000 P 822e9a368ef647bb8ba01bf4fd14ca26 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:02:38.331400 5367 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 822e9a368ef647bb8ba01bf4fd14ca26 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:02:38.331566 5367 raft_consensus.cc:686] T 00000000000000000000000000000000 P 822e9a368ef647bb8ba01bf4fd14ca26 [term 1 LEADER]: Becoming Leader. State: Replica: 822e9a368ef647bb8ba01bf4fd14ca26, State: Running, Role: LEADER
I20260927 04:02:38.331642 5364 sys_catalog.cc:564] T 00000000000000000000000000000000 P 822e9a368ef647bb8ba01bf4fd14ca26 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:02:38.331729 5367 consensus_queue.cc:237] T 00000000000000000000000000000000 P 822e9a368ef647bb8ba01bf4fd14ca26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "822e9a368ef647bb8ba01bf4fd14ca26" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 35045 } }
I20260927 04:02:38.332170 5369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 822e9a368ef647bb8ba01bf4fd14ca26 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 822e9a368ef647bb8ba01bf4fd14ca26. Latest consensus state: current_term: 1 leader_uuid: "822e9a368ef647bb8ba01bf4fd14ca26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "822e9a368ef647bb8ba01bf4fd14ca26" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 35045 } } }
I20260927 04:02:38.332165 5368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 822e9a368ef647bb8ba01bf4fd14ca26 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "822e9a368ef647bb8ba01bf4fd14ca26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "822e9a368ef647bb8ba01bf4fd14ca26" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 35045 } } }
I20260927 04:02:38.332275 5369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 822e9a368ef647bb8ba01bf4fd14ca26 [sys.catalog]: This master's current role is: LEADER
I20260927 04:02:38.332280 5368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 822e9a368ef647bb8ba01bf4fd14ca26 [sys.catalog]: This master's current role is: LEADER
I20260927 04:02:38.332528 5376 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:02:38.332820 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 5296
I20260927 04:02:38.332945 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481498018407-16101-0/minicluster-data/master-0/wal/instance
I20260927 04:02:38.333065 5376 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:02:38.333156 16101 external_mini_cluster.cc:1428] Setting key 4bf089e9a5cb1a0372e04225768b430d
I20260927 04:02:38.334947 5376 catalog_manager.cc:1347] Generated new cluster ID: c89d863ccb8e40689269abbbce936370
I20260927 04:02:38.335003 5376 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:02:38.339493 5386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:38.333617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46461 (local address 127.15.185.126:35045)
0927 04:02:38.333918 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:38.333927 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:38.333960 (+ 33us) server_negotiation.cc:407] Connection header received
0927 04:02:38.334123 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:38.334131 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:38.334191 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:38.334339 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:38.334565 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:38.336002 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:38.337100 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:38.337715 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:38.338122 (+ 407us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:38.338581 (+ 459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:38.338585 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:38.338585 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:02:38.338775 (+ 190us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:38.338810 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:02:38.338823 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:38.339016 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:38.339101 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 04:02:38.339129 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":187,"threads_started":1}
I20260927 04:02:38.340467 16101 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 04:02:38.347069 5386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:38.343353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38157 (local address 127.15.185.126:35045)
0927 04:02:38.343392 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:38.343394 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:38.343615 (+ 221us) server_negotiation.cc:407] Connection header received
0927 04:02:38.343670 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:38.343671 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:38.343681 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:38.343737 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:38.343859 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:38.344624 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:38.345699 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:38.346013 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:38.346074 (+ 61us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:38.346787 (+ 713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:38.346789 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:38.346789 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:02:38.346794 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:38.346809 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:02:38.346813 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:38.346873 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:38.346952 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 04:02:38.346963 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 04:02:38.348187 5376 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:02:38.348697 5376 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:02:38.352981 5376 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 822e9a368ef647bb8ba01bf4fd14ca26: Generated new TSK 0
I20260927 04:02:38.353147 5376 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 04:02:38.362067 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 5296
W20260927 04:02:38.366097 5281 connection.cc:535] client connection to 127.15.185.126:35045 recv error: Network error: failed to read from TLS socket (remote: 127.15.185.126:35045): Connection reset by peer (error 104)
2026-09-27T04:02:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1323 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481498018407-16101-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:02:38 dist-test-slave-glgz krb5kdc[5395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:02:38 dist-test-slave-glgz krb5kdc[5395](info): set up 2 sockets
Sep 27 04:02:38 dist-test-slave-glgz krb5kdc[5395](info): commencing operation
krb5kdc: starting...
W20260927 04:02:39.403527 16101 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Sep 27 04:02:39 dist-test-slave-glgz krb5kdc[5395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:02:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:02:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.185.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab.
Entry for principal kudu/127.15.185.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab.
I20260927 04:02:39.522182 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481498018407-16101-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481498018407-16101-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481498018407-16101-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:46343
--webserver_interface=127.15.185.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:41649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.185.126:46343
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:02:39.603919 5408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:02:39.604192 5408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:02:39.604243 5408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:02:39.605417 5408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:02:39.605494 5408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:02:39.605518 5408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:02:39.605547 5408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:02:39.605578 5408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:02:39.607107 5408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:41649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481498018407-16101-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481498018407-16101-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.185.126:46343
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:46343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481498018407-16101-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:02:39.607409 5408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:02:39.607667 5408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:02:39.612701 5413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:39.612701 5414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:39.612736 5408 server_base.cc:1034] running on GCE node
W20260927 04:02:39.612703 5416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:39.613121 5408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:02:39.613432 5408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:02:39.614614 5408 hybrid_clock.cc:648] HybridClock initialized: now 1790481759614600 us; error 52 us; skew 500 ppm
Sep 27 04:02:39 dist-test-slave-glgz krb5kdc[5395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481759, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:02:39.617188 5408 init.cc:377] Logged in from keytab as kudu/127.15.185.126@KRBTEST.COM (short username kudu)
I20260927 04:02:39.617642 5408 webserver.cc:460] Webserver started at http://127.15.185.126:45005/ using document root <none> and password file <none>
I20260927 04:02:39.617878 5408 fs_manager.cc:359] Metadata directory not provided
I20260927 04:02:39.617924 5408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:02:39.618055 5408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:02:39.619195 5408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481498018407-16101-0/minicluster-data/master-0/data/instance:
uuid: "4be122c093664dc48fd73c387e581b4f"
format_stamp: "Formatted at 2026-09-27 04:02:39 on dist-test-slave-glgz"
server_key: "d7b83a37b1333c28388c55d516b71e11"
server_key_iv: "d2f370a163d9a893c261057bf5746630"
server_key_version: "encryptionkey@0"
I20260927 04:02:39.619562 5408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481498018407-16101-0/minicluster-data/master-0/wal/instance:
uuid: "4be122c093664dc48fd73c387e581b4f"
format_stamp: "Formatted at 2026-09-27 04:02:39 on dist-test-slave-glgz"
server_key: "d7b83a37b1333c28388c55d516b71e11"
server_key_iv: "d2f370a163d9a893c261057bf5746630"
server_key_version: "encryptionkey@0"
I20260927 04:02:39.620929 5408 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 04:02:39.621701 5423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:39.621943 5408 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:02:39.622035 5408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481498018407-16101-0/minicluster-data/master-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481498018407-16101-0/minicluster-data/master-0/wal
uuid: "4be122c093664dc48fd73c387e581b4f"
format_stamp: "Formatted at 2026-09-27 04:02:39 on dist-test-slave-glgz"
server_key: "d7b83a37b1333c28388c55d516b71e11"
server_key_iv: "d2f370a163d9a893c261057bf5746630"
server_key_version: "encryptionkey@0"
I20260927 04:02:39.622090 5408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481498018407-16101-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481498018407-16101-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481498018407-16101-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:02:39.625365 5408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:02:39.627310 5408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:02:39.627416 5408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:02:39.631007 5408 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.126:46343
I20260927 04:02:39.631110 5475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.126:46343 every 8 connection(s)
I20260927 04:02:39.631422 5408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
I20260927 04:02:39.632369 5476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:02:39.634557 5476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4be122c093664dc48fd73c387e581b4f: Bootstrap starting.
I20260927 04:02:39.635180 5476 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4be122c093664dc48fd73c387e581b4f: Neither blocks nor log segments found. Creating new log.
I20260927 04:02:39.635417 5476 log.cc:824] T 00000000000000000000000000000000 P 4be122c093664dc48fd73c387e581b4f: Log is configured to *not* fsync() on all Append() calls
I20260927 04:02:39.636018 5476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4be122c093664dc48fd73c387e581b4f: No bootstrap required, opened a new log
I20260927 04:02:39.636718 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 5408
I20260927 04:02:39.636838 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481498018407-16101-0/minicluster-data/master-0/wal/instance
I20260927 04:02:39.637053 16101 external_mini_cluster.cc:1428] Setting key fd92101d9b19160212a67fff3c9d343b
I20260927 04:02:39.638150 5476 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4be122c093664dc48fd73c387e581b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4be122c093664dc48fd73c387e581b4f" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 46343 } }
I20260927 04:02:39.638330 5476 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4be122c093664dc48fd73c387e581b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:02:39.638370 5476 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4be122c093664dc48fd73c387e581b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4be122c093664dc48fd73c387e581b4f, State: Initialized, Role: FOLLOWER
I20260927 04:02:39.638549 5476 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4be122c093664dc48fd73c387e581b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4be122c093664dc48fd73c387e581b4f" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 46343 } }
I20260927 04:02:39.638667 5476 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4be122c093664dc48fd73c387e581b4f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:02:39.638725 5476 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4be122c093664dc48fd73c387e581b4f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:02:39.638780 5476 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4be122c093664dc48fd73c387e581b4f [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:02:39.639470 5480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:39.637611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38017 (local address 127.15.185.126:46343)
0927 04:02:39.637818 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:39.637830 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:39.637868 (+ 38us) server_negotiation.cc:407] Connection header received
0927 04:02:39.638108 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:39.638116 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:39.638150 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:39.638297 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:39.638933 (+ 636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:39.638937 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:39.638937 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:02:39.638942 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:39.638985 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:02:39.638999 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:39.639099 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:39.639176 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 04:02:39.639224 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":84,"threads_started":1}
I20260927 04:02:39.639858 5476 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4be122c093664dc48fd73c387e581b4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4be122c093664dc48fd73c387e581b4f" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 46343 } }
I20260927 04:02:39.639979 5476 leader_election.cc:302] T 00000000000000000000000000000000 P 4be122c093664dc48fd73c387e581b4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4be122c093664dc48fd73c387e581b4f; no voters:
I20260927 04:02:39.640136 5476 leader_election.cc:288] T 00000000000000000000000000000000 P 4be122c093664dc48fd73c387e581b4f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:02:39.640185 5481 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4be122c093664dc48fd73c387e581b4f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:02:39.640342 16101 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 04:02:39.640383 5481 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4be122c093664dc48fd73c387e581b4f [term 1 LEADER]: Becoming Leader. State: Replica: 4be122c093664dc48fd73c387e581b4f, State: Running, Role: LEADER
I20260927 04:02:39.640383 5476 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4be122c093664dc48fd73c387e581b4f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:02:39.640470 5481 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4be122c093664dc48fd73c387e581b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4be122c093664dc48fd73c387e581b4f" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 46343 } }
I20260927 04:02:39.640759 5481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4be122c093664dc48fd73c387e581b4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4be122c093664dc48fd73c387e581b4f. Latest consensus state: current_term: 1 leader_uuid: "4be122c093664dc48fd73c387e581b4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4be122c093664dc48fd73c387e581b4f" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 46343 } } }
I20260927 04:02:39.640831 5481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4be122c093664dc48fd73c387e581b4f [sys.catalog]: This master's current role is: LEADER
I20260927 04:02:39.640974 5482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4be122c093664dc48fd73c387e581b4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4be122c093664dc48fd73c387e581b4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4be122c093664dc48fd73c387e581b4f" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 46343 } } }
I20260927 04:02:39.641038 5482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4be122c093664dc48fd73c387e581b4f [sys.catalog]: This master's current role is: LEADER
W20260927 04:02:39.642302 5501 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4be122c093664dc48fd73c387e581b4f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 04:02:39.642362 5501 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 04:02:39.642418 5487 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:02:39.642599 5487 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:02:39.644006 5480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:39.641890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41389 (local address 127.15.185.126:46343)
0927 04:02:39.641924 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:39.641926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:39.642138 (+ 212us) server_negotiation.cc:407] Connection header received
0927 04:02:39.642239 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:39.642240 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:39.642247 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:39.642275 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:39.643764 (+ 1489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:39.643765 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:39.643766 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:02:39.643770 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:39.643785 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:02:39.643789 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:39.643820 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:39.643931 (+ 111us) server_negotiation.cc:299] Negotiation successful
0927 04:02:39.643945 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 04:02:39.644034 5487 catalog_manager.cc:1347] Generated new cluster ID: 856095407bb747c29d191b62a4ff0899
I20260927 04:02:39.644079 5487 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:02:39.656709 5487 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:02:39.656919 5487 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:02:39.659086 5487 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4be122c093664dc48fd73c387e581b4f: Generated new TSK 0
I20260927 04:02:39.659288 5487 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 04:02:39.663800 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 5408
2026-09-27T04:02:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1302 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481498018407-16101-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:02:39 dist-test-slave-glgz krb5kdc[5507](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:02:39 dist-test-slave-glgz krb5kdc[5507](info): set up 2 sockets
Sep 27 04:02:39 dist-test-slave-glgz krb5kdc[5507](info): commencing operation
krb5kdc: starting...
W20260927 04:02:40.707674 16101 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Sep 27 04:02:40 dist-test-slave-glgz krb5kdc[5507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:02:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:02:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.185.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab.
Entry for principal kudu/127.15.185.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab.
I20260927 04:02:40.835300 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481498018407-16101-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481498018407-16101-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481498018407-16101-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:39073
--webserver_interface=127.15.185.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:40335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.185.126:39073
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:02:40.918850 5520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:02:40.919109 5520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:02:40.919142 5520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:02:40.920449 5520 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:02:40.920499 5520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:02:40.920513 5520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:02:40.920526 5520 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:02:40.920538 5520 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:02:40.922065 5520 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:40335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481498018407-16101-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481498018407-16101-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.185.126:39073
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:39073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481498018407-16101-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:02:40.922345 5520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:02:40.922585 5520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:02:40.927794 5526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:40.927801 5528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:40.927819 5520 server_base.cc:1034] running on GCE node
W20260927 04:02:40.927801 5525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:40.928257 5520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:02:40.928567 5520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:02:40.929742 5520 hybrid_clock.cc:648] HybridClock initialized: now 1790481760929736 us; error 56 us; skew 500 ppm
Sep 27 04:02:40 dist-test-slave-glgz krb5kdc[5507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481760, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:02:40.932515 5520 init.cc:377] Logged in from keytab as kudu/127.15.185.126@KRBTEST.COM (short username kudu)
I20260927 04:02:40.932981 5520 webserver.cc:460] Webserver started at http://127.15.185.126:35451/ using document root <none> and password file <none>
I20260927 04:02:40.933238 5520 fs_manager.cc:359] Metadata directory not provided
I20260927 04:02:40.933310 5520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:02:40.933460 5520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:02:40.934581 5520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481498018407-16101-0/minicluster-data/master-0/data/instance:
uuid: "e60d822018fc4a8cac9e24526f2fc29c"
format_stamp: "Formatted at 2026-09-27 04:02:40 on dist-test-slave-glgz"
server_key: "18650f30285a66003eab64970890ba21"
server_key_iv: "7822a6e4b6cbd3c63b2622ce91a9e92d"
server_key_version: "encryptionkey@0"
I20260927 04:02:40.934958 5520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481498018407-16101-0/minicluster-data/master-0/wal/instance:
uuid: "e60d822018fc4a8cac9e24526f2fc29c"
format_stamp: "Formatted at 2026-09-27 04:02:40 on dist-test-slave-glgz"
server_key: "18650f30285a66003eab64970890ba21"
server_key_iv: "7822a6e4b6cbd3c63b2622ce91a9e92d"
server_key_version: "encryptionkey@0"
I20260927 04:02:40.936453 5520 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:02:40.937264 5535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:40.937466 5520 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:02:40.937564 5520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481498018407-16101-0/minicluster-data/master-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481498018407-16101-0/minicluster-data/master-0/wal
uuid: "e60d822018fc4a8cac9e24526f2fc29c"
format_stamp: "Formatted at 2026-09-27 04:02:40 on dist-test-slave-glgz"
server_key: "18650f30285a66003eab64970890ba21"
server_key_iv: "7822a6e4b6cbd3c63b2622ce91a9e92d"
server_key_version: "encryptionkey@0"
I20260927 04:02:40.937655 5520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481498018407-16101-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481498018407-16101-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481498018407-16101-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:02:40.949630 5520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:02:40.951804 5520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:02:40.951977 5520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:02:40.956038 5520 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.126:39073
I20260927 04:02:40.956072 5587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.126:39073 every 8 connection(s)
I20260927 04:02:40.956477 5520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
I20260927 04:02:40.957458 5588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:02:40.960291 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 5520
I20260927 04:02:40.960425 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481498018407-16101-0/minicluster-data/master-0/wal/instance
I20260927 04:02:40.960430 5588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e60d822018fc4a8cac9e24526f2fc29c: Bootstrap starting.
I20260927 04:02:40.960644 16101 external_mini_cluster.cc:1428] Setting key 324f251a02704c2a14814ebd22ba900b
I20260927 04:02:40.961203 5588 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e60d822018fc4a8cac9e24526f2fc29c: Neither blocks nor log segments found. Creating new log.
I20260927 04:02:40.961555 5588 log.cc:824] T 00000000000000000000000000000000 P e60d822018fc4a8cac9e24526f2fc29c: Log is configured to *not* fsync() on all Append() calls
I20260927 04:02:40.962296 5588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e60d822018fc4a8cac9e24526f2fc29c: No bootstrap required, opened a new log
I20260927 04:02:40.964215 5588 raft_consensus.cc:348] T 00000000000000000000000000000000 P e60d822018fc4a8cac9e24526f2fc29c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e60d822018fc4a8cac9e24526f2fc29c" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 39073 } }
I20260927 04:02:40.964385 5588 raft_consensus.cc:374] T 00000000000000000000000000000000 P e60d822018fc4a8cac9e24526f2fc29c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:02:40.964432 5588 raft_consensus.cc:729] T 00000000000000000000000000000000 P e60d822018fc4a8cac9e24526f2fc29c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e60d822018fc4a8cac9e24526f2fc29c, State: Initialized, Role: FOLLOWER
I20260927 04:02:40.964573 5588 consensus_queue.cc:260] T 00000000000000000000000000000000 P e60d822018fc4a8cac9e24526f2fc29c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e60d822018fc4a8cac9e24526f2fc29c" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 39073 } }
I20260927 04:02:40.964674 5588 raft_consensus.cc:388] T 00000000000000000000000000000000 P e60d822018fc4a8cac9e24526f2fc29c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:02:40.964735 5588 raft_consensus.cc:482] T 00000000000000000000000000000000 P e60d822018fc4a8cac9e24526f2fc29c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:02:40.964790 5588 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e60d822018fc4a8cac9e24526f2fc29c [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:02:40.965382 5588 raft_consensus.cc:504] T 00000000000000000000000000000000 P e60d822018fc4a8cac9e24526f2fc29c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e60d822018fc4a8cac9e24526f2fc29c" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 39073 } }
I20260927 04:02:40.965503 5588 leader_election.cc:302] T 00000000000000000000000000000000 P e60d822018fc4a8cac9e24526f2fc29c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e60d822018fc4a8cac9e24526f2fc29c; no voters:
I20260927 04:02:40.965708 5588 leader_election.cc:288] T 00000000000000000000000000000000 P e60d822018fc4a8cac9e24526f2fc29c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:02:40.965867 5594 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e60d822018fc4a8cac9e24526f2fc29c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:02:40.965991 5588 sys_catalog.cc:564] T 00000000000000000000000000000000 P e60d822018fc4a8cac9e24526f2fc29c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:02:40.966028 5594 raft_consensus.cc:686] T 00000000000000000000000000000000 P e60d822018fc4a8cac9e24526f2fc29c [term 1 LEADER]: Becoming Leader. State: Replica: e60d822018fc4a8cac9e24526f2fc29c, State: Running, Role: LEADER
I20260927 04:02:40.966193 5594 consensus_queue.cc:237] T 00000000000000000000000000000000 P e60d822018fc4a8cac9e24526f2fc29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e60d822018fc4a8cac9e24526f2fc29c" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 39073 } }
I20260927 04:02:40.966684 5595 sys_catalog.cc:455] T 00000000000000000000000000000000 P e60d822018fc4a8cac9e24526f2fc29c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e60d822018fc4a8cac9e24526f2fc29c. Latest consensus state: current_term: 1 leader_uuid: "e60d822018fc4a8cac9e24526f2fc29c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e60d822018fc4a8cac9e24526f2fc29c" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 39073 } } }
I20260927 04:02:40.966789 5595 sys_catalog.cc:458] T 00000000000000000000000000000000 P e60d822018fc4a8cac9e24526f2fc29c [sys.catalog]: This master's current role is: LEADER
I20260927 04:02:40.966987 5593 sys_catalog.cc:455] T 00000000000000000000000000000000 P e60d822018fc4a8cac9e24526f2fc29c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e60d822018fc4a8cac9e24526f2fc29c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e60d822018fc4a8cac9e24526f2fc29c" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 39073 } } }
I20260927 04:02:40.967109 5593 sys_catalog.cc:458] T 00000000000000000000000000000000 P e60d822018fc4a8cac9e24526f2fc29c [sys.catalog]: This master's current role is: LEADER
I20260927 04:02:40.967024 5602 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:02:40.967773 5602 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 04:02:40 dist-test-slave-glgz krb5kdc[5507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 04:02:40.972322 5591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:40.961237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54539 (local address 127.15.185.126:39073)
0927 04:02:40.961449 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:40.961460 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:40.961491 (+ 31us) server_negotiation.cc:407] Connection header received
0927 04:02:40.961976 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:40.961983 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:40.962032 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:40.962190 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:40.962873 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:40.963856 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:40.964738 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:40.965232 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:40.965356 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:40.968603 (+ 3247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:40.968608 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:40.968608 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:40.968621 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:40.970352 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:40.970741 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:40.970744 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:40.970745 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:40.970849 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:40.971107 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:40.971108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:40.971109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:40.971437 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:40.971602 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:40.971890 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 04:02:40.972067 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":85,"threads_started":1}
I20260927 04:02:40.972800 5602 catalog_manager.cc:1347] Generated new cluster ID: 6ec65ba8b8694e61b928ead523097d00
I20260927 04:02:40.972851 5602 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:02:40.973749 16101 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 04:02:40.981421 5602 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:02:40.982077 5602 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:02:40.983160 5591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:40.975396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55677 (local address 127.15.185.126:39073)
0927 04:02:40.975437 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:40.975439 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:40.975448 (+ 9us) server_negotiation.cc:407] Connection header received
0927 04:02:40.975473 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:40.975474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:40.975485 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:40.975561 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:40.976232 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:40.977006 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:40.977982 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:40.978315 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:40.978392 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:40.979839 (+ 1447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:40.979841 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:40.979841 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:40.979848 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:40.981887 (+ 2039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:40.982314 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:40.982315 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:40.982315 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:40.982376 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:40.982567 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:40.982568 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:40.982568 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:40.982695 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:40.982806 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:40.982920 (+ 114us) server_negotiation.cc:299] Negotiation successful
0927 04:02:40.983026 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":21}
I20260927 04:02:40.986452 5602 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e60d822018fc4a8cac9e24526f2fc29c: Generated new TSK 0
I20260927 04:02:40.986650 5602 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 04:02:40.989791 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 5520
2026-09-27T04:02:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1325 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481498018407-16101-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:02:41 dist-test-slave-glgz krb5kdc[5619](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:02:41 dist-test-slave-glgz krb5kdc[5619](info): set up 2 sockets
Sep 27 04:02:41 dist-test-slave-glgz krb5kdc[5619](info): commencing operation
krb5kdc: starting...
W20260927 04:02:42.043808 16101 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Sep 27 04:02:42 dist-test-slave-glgz krb5kdc[5619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:02:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:02:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.185.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab.
Entry for principal kudu/127.15.185.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab.
I20260927 04:02:42.172142 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481498018407-16101-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481498018407-16101-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481498018407-16101-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:38239
--webserver_interface=127.15.185.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:38705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.185.126:38239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:02:42.257542 5632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:02:42.257830 5632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:02:42.257860 5632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:02:42.259303 5632 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:02:42.259354 5632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:02:42.259369 5632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:02:42.259383 5632 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:02:42.259395 5632 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:02:42.261075 5632 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:38705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481498018407-16101-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481498018407-16101-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.185.126:38239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:38239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481498018407-16101-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:02:42.261389 5632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:02:42.261648 5632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:02:42.267063 5638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:42.267063 5640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:42.267110 5637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:42.267360 5632 server_base.cc:1034] running on GCE node
I20260927 04:02:42.267665 5632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:02:42.268002 5632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:02:42.269198 5632 hybrid_clock.cc:648] HybridClock initialized: now 1790481762269172 us; error 40 us; skew 500 ppm
Sep 27 04:02:42 dist-test-slave-glgz krb5kdc[5619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481762, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:02:42.276598 5632 init.cc:377] Logged in from keytab as kudu/127.15.185.126@KRBTEST.COM (short username kudu)
I20260927 04:02:42.277249 5632 webserver.cc:460] Webserver started at http://127.15.185.126:39419/ using document root <none> and password file <none>
I20260927 04:02:42.277504 5632 fs_manager.cc:359] Metadata directory not provided
I20260927 04:02:42.277576 5632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:02:42.277742 5632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:02:42.278905 5632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481498018407-16101-0/minicluster-data/master-0/data/instance:
uuid: "91f87daf95824a7096e6513c0062f334"
format_stamp: "Formatted at 2026-09-27 04:02:42 on dist-test-slave-glgz"
server_key: "b88a6ff1b67360f3e44451fc83e1d909"
server_key_iv: "fb097d5cd51d50ba4e7f26997f568f43"
server_key_version: "encryptionkey@0"
I20260927 04:02:42.279371 5632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481498018407-16101-0/minicluster-data/master-0/wal/instance:
uuid: "91f87daf95824a7096e6513c0062f334"
format_stamp: "Formatted at 2026-09-27 04:02:42 on dist-test-slave-glgz"
server_key: "b88a6ff1b67360f3e44451fc83e1d909"
server_key_iv: "fb097d5cd51d50ba4e7f26997f568f43"
server_key_version: "encryptionkey@0"
I20260927 04:02:42.280937 5632 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:02:42.281950 5647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:42.282236 5632 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:02:42.282377 5632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481498018407-16101-0/minicluster-data/master-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481498018407-16101-0/minicluster-data/master-0/wal
uuid: "91f87daf95824a7096e6513c0062f334"
format_stamp: "Formatted at 2026-09-27 04:02:42 on dist-test-slave-glgz"
server_key: "b88a6ff1b67360f3e44451fc83e1d909"
server_key_iv: "fb097d5cd51d50ba4e7f26997f568f43"
server_key_version: "encryptionkey@0"
I20260927 04:02:42.282469 5632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481498018407-16101-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481498018407-16101-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481498018407-16101-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:02:42.297778 5632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:02:42.300489 5632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:02:42.300694 5632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:02:42.305065 5632 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.126:38239
I20260927 04:02:42.305101 5699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.126:38239 every 8 connection(s)
I20260927 04:02:42.305612 5632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
I20260927 04:02:42.306900 5700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:02:42.307600 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 5632
I20260927 04:02:42.307753 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481498018407-16101-0/minicluster-data/master-0/wal/instance
I20260927 04:02:42.307974 16101 external_mini_cluster.cc:1428] Setting key 92a045db9c594ad9ce6e7bd6a9cbf323
I20260927 04:02:42.310287 5700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91f87daf95824a7096e6513c0062f334: Bootstrap starting.
I20260927 04:02:42.311018 5700 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91f87daf95824a7096e6513c0062f334: Neither blocks nor log segments found. Creating new log.
I20260927 04:02:42.311401 5700 log.cc:824] T 00000000000000000000000000000000 P 91f87daf95824a7096e6513c0062f334: Log is configured to *not* fsync() on all Append() calls
I20260927 04:02:42.312141 5700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91f87daf95824a7096e6513c0062f334: No bootstrap required, opened a new log
I20260927 04:02:42.314034 5703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:42.308714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57013 (local address 127.15.185.126:38239)
0927 04:02:42.308909 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:42.308919 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:42.308949 (+ 30us) server_negotiation.cc:407] Connection header received
0927 04:02:42.309418 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:42.309426 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:42.309479 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:42.309606 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:42.309889 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:42.310965 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:42.311900 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:42.312408 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:42.312539 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:42.313326 (+ 787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:42.313331 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:42.313331 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:02:42.313524 (+ 193us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:42.313557 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:02:42.313575 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:42.313661 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:42.313745 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 04:02:42.313828 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":99,"threads_started":1}
I20260927 04:02:42.314553 5700 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91f87daf95824a7096e6513c0062f334 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91f87daf95824a7096e6513c0062f334" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 38239 } }
I20260927 04:02:42.314778 5700 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91f87daf95824a7096e6513c0062f334 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:02:42.314859 5700 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91f87daf95824a7096e6513c0062f334 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91f87daf95824a7096e6513c0062f334, State: Initialized, Role: FOLLOWER
I20260927 04:02:42.315030 5700 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91f87daf95824a7096e6513c0062f334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91f87daf95824a7096e6513c0062f334" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 38239 } }
I20260927 04:02:42.315178 5700 raft_consensus.cc:388] T 00000000000000000000000000000000 P 91f87daf95824a7096e6513c0062f334 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:02:42.315243 5700 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91f87daf95824a7096e6513c0062f334 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:02:42.315320 5700 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91f87daf95824a7096e6513c0062f334 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:02:42.315330 16101 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 04:02:42.315990 5700 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91f87daf95824a7096e6513c0062f334 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91f87daf95824a7096e6513c0062f334" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 38239 } }
I20260927 04:02:42.316126 5700 leader_election.cc:302] T 00000000000000000000000000000000 P 91f87daf95824a7096e6513c0062f334 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 91f87daf95824a7096e6513c0062f334; no voters:
I20260927 04:02:42.316370 5700 leader_election.cc:288] T 00000000000000000000000000000000 P 91f87daf95824a7096e6513c0062f334 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:02:42.316427 5707 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91f87daf95824a7096e6513c0062f334 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:02:42.316550 5707 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91f87daf95824a7096e6513c0062f334 [term 1 LEADER]: Becoming Leader. State: Replica: 91f87daf95824a7096e6513c0062f334, State: Running, Role: LEADER
I20260927 04:02:42.316646 5707 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91f87daf95824a7096e6513c0062f334 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91f87daf95824a7096e6513c0062f334" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 38239 } }
I20260927 04:02:42.316670 5700 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91f87daf95824a7096e6513c0062f334 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:02:42.317085 5707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91f87daf95824a7096e6513c0062f334 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91f87daf95824a7096e6513c0062f334. Latest consensus state: current_term: 1 leader_uuid: "91f87daf95824a7096e6513c0062f334" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91f87daf95824a7096e6513c0062f334" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 38239 } } }
I20260927 04:02:42.317096 5710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91f87daf95824a7096e6513c0062f334 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "91f87daf95824a7096e6513c0062f334" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91f87daf95824a7096e6513c0062f334" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 38239 } } }
I20260927 04:02:42.317234 5710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91f87daf95824a7096e6513c0062f334 [sys.catalog]: This master's current role is: LEADER
I20260927 04:02:42.317165 5707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91f87daf95824a7096e6513c0062f334 [sys.catalog]: This master's current role is: LEADER
I20260927 04:02:42.317790 5717 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:02:42.318251 5717 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:02:42.320262 5717 catalog_manager.cc:1347] Generated new cluster ID: 9baa02d60fc3430188fbd39bcfaaa7ee
I20260927 04:02:42.320315 5717 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:02:42.322116 5703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:42.316742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33201 (local address 127.15.185.126:38239)
0927 04:02:42.316784 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:42.316786 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:42.318699 (+ 1913us) server_negotiation.cc:407] Connection header received
0927 04:02:42.318799 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:42.318800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:42.318812 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:42.318866 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:42.319074 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:42.319797 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:42.320664 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:42.320940 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:42.321043 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:42.321857 (+ 814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:42.321858 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:42.321859 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:02:42.321864 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:42.321879 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:02:42.321884 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:42.321935 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:42.321996 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 04:02:42.322023 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 04:02:42.341045 5717 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:02:42.341816 5717 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:02:42.345218 5717 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 91f87daf95824a7096e6513c0062f334: Generated new TSK 0
I20260927 04:02:42.345422 5717 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 04:02:42.354084 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 5632
2026-09-27T04:02:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1364 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481498018407-16101-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:02:42 dist-test-slave-glgz krb5kdc[5731](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:02:42 dist-test-slave-glgz krb5kdc[5731](info): set up 2 sockets
Sep 27 04:02:42 dist-test-slave-glgz krb5kdc[5731](info): commencing operation
krb5kdc: starting...
W20260927 04:02:43.427644 16101 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Sep 27 04:02:43 dist-test-slave-glgz krb5kdc[5731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:02:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:02:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.185.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab.
Entry for principal kudu/127.15.185.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab.
I20260927 04:02:43.549965 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481498018407-16101-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481498018407-16101-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481498018407-16101-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:39773
--webserver_interface=127.15.185.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:42307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.185.126:39773
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:02:43.632762 5744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:02:43.633029 5744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:02:43.633069 5744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:02:43.634307 5744 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:02:43.634387 5744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:02:43.634414 5744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:02:43.634444 5744 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:02:43.634465 5744 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:02:43.636008 5744 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:42307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481498018407-16101-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481498018407-16101-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.185.126:39773
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:39773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481498018407-16101-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:02:43.636310 5744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:02:43.636561 5744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:02:43.641641 5752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:43.641731 5749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:43.641655 5750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:43.642247 5744 server_base.cc:1034] running on GCE node
I20260927 04:02:43.642464 5744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:02:43.642729 5744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:02:43.643898 5744 hybrid_clock.cc:648] HybridClock initialized: now 1790481763643881 us; error 41 us; skew 500 ppm
Sep 27 04:02:43 dist-test-slave-glgz krb5kdc[5731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481763, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:02:43.646876 5744 init.cc:377] Logged in from keytab as kudu/127.15.185.126@KRBTEST.COM (short username kudu)
I20260927 04:02:43.647486 5744 webserver.cc:460] Webserver started at http://127.15.185.126:44915/ using document root <none> and password file <none>
I20260927 04:02:43.647734 5744 fs_manager.cc:359] Metadata directory not provided
I20260927 04:02:43.647804 5744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:02:43.648015 5744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:02:43.649276 5744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481498018407-16101-0/minicluster-data/master-0/data/instance:
uuid: "51ff5b31a3e948198ec7b131000ff432"
format_stamp: "Formatted at 2026-09-27 04:02:43 on dist-test-slave-glgz"
server_key: "1ef5bf419ca3be170ba0f04df0b55850"
server_key_iv: "a4abe0258bf364563c632c527931b3f9"
server_key_version: "encryptionkey@0"
I20260927 04:02:43.649922 5744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481498018407-16101-0/minicluster-data/master-0/wal/instance:
uuid: "51ff5b31a3e948198ec7b131000ff432"
format_stamp: "Formatted at 2026-09-27 04:02:43 on dist-test-slave-glgz"
server_key: "1ef5bf419ca3be170ba0f04df0b55850"
server_key_iv: "a4abe0258bf364563c632c527931b3f9"
server_key_version: "encryptionkey@0"
I20260927 04:02:43.651847 5744 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260927 04:02:43.652843 5759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:43.653106 5744 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:02:43.653234 5744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481498018407-16101-0/minicluster-data/master-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481498018407-16101-0/minicluster-data/master-0/wal
uuid: "51ff5b31a3e948198ec7b131000ff432"
format_stamp: "Formatted at 2026-09-27 04:02:43 on dist-test-slave-glgz"
server_key: "1ef5bf419ca3be170ba0f04df0b55850"
server_key_iv: "a4abe0258bf364563c632c527931b3f9"
server_key_version: "encryptionkey@0"
I20260927 04:02:43.653381 5744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481498018407-16101-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481498018407-16101-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481498018407-16101-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:02:43.657084 5744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:02:43.659302 5744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:02:43.659461 5744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:02:43.663693 5744 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.126:39773
I20260927 04:02:43.663735 5811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.126:39773 every 8 connection(s)
I20260927 04:02:43.664096 5744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
I20260927 04:02:43.665057 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 5744
I20260927 04:02:43.665138 5812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:02:43.665221 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481498018407-16101-0/minicluster-data/master-0/wal/instance
I20260927 04:02:43.665431 16101 external_mini_cluster.cc:1428] Setting key 34df956bb689943d218ada67da9f727a
I20260927 04:02:43.668268 5815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:43.666052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50909 (local address 127.15.185.126:39773)
0927 04:02:43.666237 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:43.666244 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:43.666271 (+ 27us) server_negotiation.cc:407] Connection header received
0927 04:02:43.666725 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:43.666731 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:43.666757 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:43.666877 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:43.667748 (+ 871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:43.667751 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:43.667752 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:02:43.667756 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:43.667810 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:02:43.667820 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:43.667885 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:43.667977 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 04:02:43.668041 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":89,"threads_started":1}
I20260927 04:02:43.668792 5812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51ff5b31a3e948198ec7b131000ff432: Bootstrap starting.
I20260927 04:02:43.669275 16101 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 04:02:43.669509 5812 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51ff5b31a3e948198ec7b131000ff432: Neither blocks nor log segments found. Creating new log.
I20260927 04:02:43.669821 5812 log.cc:824] T 00000000000000000000000000000000 P 51ff5b31a3e948198ec7b131000ff432: Log is configured to *not* fsync() on all Append() calls
I20260927 04:02:43.670683 5812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51ff5b31a3e948198ec7b131000ff432: No bootstrap required, opened a new log
I20260927 04:02:43.671973 5815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:43.670967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41965 (local address 127.15.185.126:39773)
0927 04:02:43.671020 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:43.671022 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:43.671031 (+ 9us) server_negotiation.cc:407] Connection header received
0927 04:02:43.671121 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:43.671124 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:43.671139 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:43.671185 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:43.671754 (+ 569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:43.671755 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:43.671755 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:02:43.671759 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:43.671772 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:02:43.671775 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:43.671820 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:43.671873 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 04:02:43.671884 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 04:02:43.673265 5812 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51ff5b31a3e948198ec7b131000ff432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51ff5b31a3e948198ec7b131000ff432" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 39773 } }
I20260927 04:02:43.673439 5812 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51ff5b31a3e948198ec7b131000ff432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:02:43.673486 5812 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51ff5b31a3e948198ec7b131000ff432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51ff5b31a3e948198ec7b131000ff432, State: Initialized, Role: FOLLOWER
I20260927 04:02:43.673636 5812 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51ff5b31a3e948198ec7b131000ff432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51ff5b31a3e948198ec7b131000ff432" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 39773 } }
I20260927 04:02:43.673735 5812 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51ff5b31a3e948198ec7b131000ff432 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:02:43.673780 5812 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51ff5b31a3e948198ec7b131000ff432 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:02:43.673836 5812 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51ff5b31a3e948198ec7b131000ff432 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:02:43.674424 5812 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51ff5b31a3e948198ec7b131000ff432 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51ff5b31a3e948198ec7b131000ff432" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 39773 } }
I20260927 04:02:43.674556 5812 leader_election.cc:302] T 00000000000000000000000000000000 P 51ff5b31a3e948198ec7b131000ff432 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51ff5b31a3e948198ec7b131000ff432; no voters:
I20260927 04:02:43.674731 5812 leader_election.cc:288] T 00000000000000000000000000000000 P 51ff5b31a3e948198ec7b131000ff432 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:02:43.674777 5822 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51ff5b31a3e948198ec7b131000ff432 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:02:43.674990 5812 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51ff5b31a3e948198ec7b131000ff432 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:02:43.674998 5822 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51ff5b31a3e948198ec7b131000ff432 [term 1 LEADER]: Becoming Leader. State: Replica: 51ff5b31a3e948198ec7b131000ff432, State: Running, Role: LEADER
I20260927 04:02:43.675222 5822 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51ff5b31a3e948198ec7b131000ff432 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51ff5b31a3e948198ec7b131000ff432" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 39773 } }
I20260927 04:02:43.675843 5823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51ff5b31a3e948198ec7b131000ff432 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "51ff5b31a3e948198ec7b131000ff432" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51ff5b31a3e948198ec7b131000ff432" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 39773 } } }
I20260927 04:02:43.675976 5823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51ff5b31a3e948198ec7b131000ff432 [sys.catalog]: This master's current role is: LEADER
I20260927 04:02:43.675846 5824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51ff5b31a3e948198ec7b131000ff432 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51ff5b31a3e948198ec7b131000ff432. Latest consensus state: current_term: 1 leader_uuid: "51ff5b31a3e948198ec7b131000ff432" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51ff5b31a3e948198ec7b131000ff432" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 39773 } } }
I20260927 04:02:43.676057 5824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51ff5b31a3e948198ec7b131000ff432 [sys.catalog]: This master's current role is: LEADER
I20260927 04:02:43.676440 5831 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:02:43.676951 5831 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:02:43.678552 5831 catalog_manager.cc:1347] Generated new cluster ID: 142f7eb4d4b34842b1501f6fcc5f6aec
I20260927 04:02:43.678642 5831 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:02:43.694605 5831 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:02:43.694842 5831 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:02:43.700973 5831 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 51ff5b31a3e948198ec7b131000ff432: Generated new TSK 0
I20260927 04:02:43.701295 5831 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 04:02:43.714707 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 5744
2026-09-27T04:02:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1362 ms)
[----------] 18 tests from AuthTokenIssuingTest (24119 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:02:43 dist-test-slave-glgz krb5kdc[5843](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:02:43 dist-test-slave-glgz krb5kdc[5843](info): set up 2 sockets
Sep 27 04:02:43 dist-test-slave-glgz krb5kdc[5843](info): commencing operation
krb5kdc: starting...
W20260927 04:02:44.759836 16101 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Sep 27 04:02:44 dist-test-slave-glgz krb5kdc[5843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:02:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:02:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.185.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab.
Entry for principal kudu/127.15.185.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab.
I20260927 04:02:44.891268 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:45777
--webserver_interface=127.15.185.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:40951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.185.126:45777,127.15.185.125:45391,127.15.185.124:42753
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:02:44.976074 5856 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 04:02:44.976363 5856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:02:44.976397 5856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:02:44.976442 5856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:02:44.977679 5856 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:02:44.977754 5856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:02:44.977778 5856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:02:44.977807 5856 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:02:44.977824 5856 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:02:44.979521 5856 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:40951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.185.126:45777,127.15.185.125:45391,127.15.185.124:42753
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:45777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:02:44.979835 5856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:02:44.980086 5856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:02:44.985387 5861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:44.985378 5862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:44.985407 5856 server_base.cc:1034] running on GCE node
W20260927 04:02:44.985357 5864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:44.985921 5856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:02:44.986203 5856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:02:44.987373 5856 hybrid_clock.cc:648] HybridClock initialized: now 1790481764987359 us; error 47 us; skew 500 ppm
Sep 27 04:02:44 dist-test-slave-glgz krb5kdc[5843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481764, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:02:44.989998 5856 init.cc:377] Logged in from keytab as kudu/127.15.185.126@KRBTEST.COM (short username kudu)
I20260927 04:02:44.990502 5856 webserver.cc:460] Webserver started at http://127.15.185.126:41703/ using document root <none> and password file <none>
I20260927 04:02:44.990749 5856 fs_manager.cc:359] Metadata directory not provided
I20260927 04:02:44.990796 5856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:02:44.990890 5856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:02:44.992015 5856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/minicluster-data/master-0/data/instance:
uuid: "d0117710d7e544188d814baa51cf1230"
format_stamp: "Formatted at 2026-09-27 04:02:44 on dist-test-slave-glgz"
server_key: "306654360bced5c2df302829eed06bcd"
server_key_iv: "bb9e42cf1aa1b5303ff72ab9b2a5e372"
server_key_version: "encryptionkey@0"
I20260927 04:02:44.992322 5856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/minicluster-data/master-0/wal/instance:
uuid: "d0117710d7e544188d814baa51cf1230"
format_stamp: "Formatted at 2026-09-27 04:02:44 on dist-test-slave-glgz"
server_key: "306654360bced5c2df302829eed06bcd"
server_key_iv: "bb9e42cf1aa1b5303ff72ab9b2a5e372"
server_key_version: "encryptionkey@0"
I20260927 04:02:44.993685 5856 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 04:02:44.994462 5871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:44.994735 5856 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:02:44.994979 5856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/minicluster-data/master-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/minicluster-data/master-0/wal
uuid: "d0117710d7e544188d814baa51cf1230"
format_stamp: "Formatted at 2026-09-27 04:02:44 on dist-test-slave-glgz"
server_key: "306654360bced5c2df302829eed06bcd"
server_key_iv: "bb9e42cf1aa1b5303ff72ab9b2a5e372"
server_key_version: "encryptionkey@0"
I20260927 04:02:44.995103 5856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:02:45.012120 5856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:02:45.014792 5856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:02:45.014981 5856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:02:45.019187 5856 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.126:45777
I20260927 04:02:45.019241 5923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.126:45777 every 8 connection(s)
I20260927 04:02:45.019644 5856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
I20260927 04:02:45.020602 5924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:02:45.022341 5924 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45777 } has no permanent_uuid. Determining permanent_uuid...
I20260927 04:02:45.027245 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 5856
I20260927 04:02:45.027372 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/minicluster-data/master-0/wal/instance
I20260927 04:02:45.027592 16101 external_mini_cluster.cc:1428] Setting key 1a4c7e1c21e4ffe8f51a0203c4fa41e7
Sep 27 04:02:45 dist-test-slave-glgz krb5kdc[5843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481764, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.126@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 04:02:45.032997 5927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:45.023157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43340 (local address 127.15.185.126:45777)
0927 04:02:45.023363 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:45.023373 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:45.023531 (+ 158us) server_negotiation.cc:407] Connection header received
0927 04:02:45.024218 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:45.024220 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:45.024238 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:45.024341 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:45.024935 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:45.025770 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:45.026668 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:45.026975 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:45.027195 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:45.029543 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:45.029551 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:45.029551 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:45.029558 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:45.031294 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:45.031782 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:45.031784 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:45.031784 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:45.031861 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:45.032158 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:45.032158 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:45.032159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:45.032368 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:45.032431 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:45.032696 (+ 265us) server_negotiation.cc:299] Negotiation successful
0927 04:02:45.032828 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":148,"thread_start_us":109,"threads_started":1}
I20260927 04:02:45.033075 5926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:45.023146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45777 (local address 127.0.0.1:43340)
0927 04:02:45.023372 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:45.023414 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:45.024016 (+ 602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:45.024346 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:45.024349 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:45.024668 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:02:45.024829 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:45.024834 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:45.025868 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:45.025869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:45.026555 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:45.026557 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:45.027092 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:45.027093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:45.027195 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:45.027848 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:02:45.027857 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:02:45.029426 (+ 1569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:02:45.031378 (+ 1952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:45.031385 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:45.031388 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:45.031694 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:45.031936 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:45.031937 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:45.031937 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:45.032061 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:45.032444 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:02:45.032452 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:02:45.032541 (+ 89us) client_negotiation.cc:769] Sending connection context
0927 04:02:45.032669 (+ 128us) client_negotiation.cc:240] Negotiation successful
0927 04:02:45.032915 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":64,"thread_start_us":114,"threads_started":1}
I20260927 04:02:45.033869 5924 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.185.125" port: 45391 } has no permanent_uuid. Determining permanent_uuid...
I20260927 04:02:45.034247 5926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:45.034110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.125:45391 (local address 127.0.0.1:44060)
0927 04:02:45.034170 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:45.034222 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.125:45391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 04:02:45.034426 5874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.185.125:45391: connect: Connection refused (error 111)
I20260927 04:02:45.034883 5926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:45.034766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.125:45391 (local address 127.0.0.1:44066)
0927 04:02:45.034822 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:45.034853 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.125:45391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260927 04:02:45.035082 5924 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.185.125:45391: Network error: Client connection negotiation failed: client connection to 127.15.185.125:45391: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.15.185.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.125.keytab.
Entry for principal kudu/127.15.185.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.125.keytab.
I20260927 04:02:45.055438 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.125:45391
--webserver_interface=127.15.185.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.125.keytab
--principal=kudu/127.15.185.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:40951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.185.126:45777,127.15.185.125:45391,127.15.185.124:42753
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 04:02:45.084347 5924 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.185.125" port: 45391 } attempt: 1
I20260927 04:02:45.084937 5926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:45.084772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.125:45391 (local address 127.0.0.1:44068)
0927 04:02:45.084844 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:45.084897 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.125:45391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 04:02:45.085415 5926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:45.085312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.125:45391 (local address 127.0.0.1:44080)
0927 04:02:45.085362 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:45.085393 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.125:45391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260927 04:02:45.085626 5924 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.185.125:45391: Network error: Client connection negotiation failed: client connection to 127.15.185.125:45391: connect: Connection refused (error 111)
W20260927 04:02:45.145928 5931 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 04:02:45.146178 5931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:02:45.146196 5931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:02:45.146217 5931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:02:45.147580 5931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:02:45.147632 5931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:02:45.147646 5931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:02:45.147660 5931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:02:45.147671 5931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:02:45.149230 5931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:40951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.15.185.126:45777,127.15.185.125:45391,127.15.185.124:42753
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.125.keytab
--principal=kudu/127.15.185.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.125:45391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.15.185.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:02:45.149497 5931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:02:45.149745 5931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 04:02:45.153947 5924 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.185.125" port: 45391 } attempt: 2
I20260927 04:02:45.154482 5926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:45.154318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.125:45391 (local address 127.0.0.1:44086)
0927 04:02:45.154394 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:45.154434 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.125:45391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 04:02:45.154944 5936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:45.155084 5926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:45.154971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.125:45391 (local address 127.0.0.1:44098)
0927 04:02:45.155011 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:45.155059 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.125:45391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260927 04:02:45.155112 5937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:45.154950 5939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:45.155202 5924 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.185.125:45391: Network error: Client connection negotiation failed: client connection to 127.15.185.125:45391: connect: Connection refused (error 111)
I20260927 04:02:45.155412 5931 server_base.cc:1034] running on GCE node
I20260927 04:02:45.155669 5931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:02:45.155983 5931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:02:45.157171 5931 hybrid_clock.cc:648] HybridClock initialized: now 1790481765157145 us; error 39 us; skew 500 ppm
Sep 27 04:02:45 dist-test-slave-glgz krb5kdc[5843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481765, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:02:45.159830 5931 init.cc:377] Logged in from keytab as kudu/127.15.185.125@KRBTEST.COM (short username kudu)
I20260927 04:02:45.160343 5931 webserver.cc:460] Webserver started at http://127.15.185.125:37785/ using document root <none> and password file <none>
I20260927 04:02:45.160615 5931 fs_manager.cc:359] Metadata directory not provided
I20260927 04:02:45.160661 5931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:02:45.160753 5931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:02:45.161791 5931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/minicluster-data/master-1/data/instance:
uuid: "6a62c49c62384f649489cb5dc8d9dda8"
format_stamp: "Formatted at 2026-09-27 04:02:45 on dist-test-slave-glgz"
server_key: "f381170b48a03849c1d7fef9cef4643c"
server_key_iv: "c09d9a7e20b60f87f1727faeeda1c608"
server_key_version: "encryptionkey@0"
I20260927 04:02:45.162092 5931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/minicluster-data/master-1/wal/instance:
uuid: "6a62c49c62384f649489cb5dc8d9dda8"
format_stamp: "Formatted at 2026-09-27 04:02:45 on dist-test-slave-glgz"
server_key: "f381170b48a03849c1d7fef9cef4643c"
server_key_iv: "c09d9a7e20b60f87f1727faeeda1c608"
server_key_version: "encryptionkey@0"
I20260927 04:02:45.163427 5931 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 04:02:45.164232 5946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:45.164422 5931 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:02:45.164530 5931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/minicluster-data/master-1/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/minicluster-data/master-1/wal
uuid: "6a62c49c62384f649489cb5dc8d9dda8"
format_stamp: "Formatted at 2026-09-27 04:02:45 on dist-test-slave-glgz"
server_key: "f381170b48a03849c1d7fef9cef4643c"
server_key_iv: "c09d9a7e20b60f87f1727faeeda1c608"
server_key_version: "encryptionkey@0"
I20260927 04:02:45.164671 5931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:02:45.172919 5931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:02:45.175209 5931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:02:45.175392 5931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:02:45.179759 5931 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.125:45391
I20260927 04:02:45.179788 5998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.125:45391 every 8 connection(s)
I20260927 04:02:45.180161 5931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/minicluster-data/master-1/data/info.pb
I20260927 04:02:45.180842 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 5931
I20260927 04:02:45.180962 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/minicluster-data/master-1/wal/instance
I20260927 04:02:45.181069 5999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:02:45.181196 16101 external_mini_cluster.cc:1428] Setting key d9ab3d21628a1263ebfdd4d3e4de4e16
I20260927 04:02:45.182722 5999 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45777 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.15.185.124@KRBTEST.COM; defaulting to no policy
Sep 27 04:02:45 dist-test-slave-glgz krb5kdc[5843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481765, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.125@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.124@KRBTEST.COM" created.
I20260927 04:02:45.193228 5927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:45.183546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43348 (local address 127.15.185.126:45777)
0927 04:02:45.183619 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:45.183622 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:45.183803 (+ 181us) server_negotiation.cc:407] Connection header received
0927 04:02:45.184289 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:45.184291 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:45.184305 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:45.184385 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:45.185030 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:45.185877 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:45.186900 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:45.187303 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:45.187391 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:45.189875 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:45.189876 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:45.189877 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:45.189884 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:45.191381 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:45.192058 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:45.192060 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:45.192060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:45.192120 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:45.192391 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:45.192392 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:45.192392 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:45.192551 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:45.192625 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:45.193001 (+ 376us) server_negotiation.cc:299] Negotiation successful
0927 04:02:45.193100 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 04:02:45.193442 6002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:45.183514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45777 (local address 127.0.0.1:43348)
0927 04:02:45.183669 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:45.183712 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:45.184191 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:45.184461 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:45.184465 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:45.184728 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:02:45.184931 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:45.184935 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:45.185990 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:45.185991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:45.186797 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:45.186799 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:45.187402 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:45.187403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:45.187519 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:45.188185 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:02:45.188194 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:02:45.189620 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:02:45.191715 (+ 2095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:45.191724 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:45.191726 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:45.191976 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:45.192217 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:45.192217 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:45.192217 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:45.192333 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:45.192632 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:02:45.192698 (+ 66us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:02:45.192841 (+ 143us) client_negotiation.cc:769] Sending connection context
0927 04:02:45.192986 (+ 145us) client_negotiation.cc:240] Negotiation successful
0927 04:02:45.193169 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":77,"threads_started":1}
I20260927 04:02:45.194085 5999 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.185.125" port: 45391 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 04:02:45 dist-test-slave-glgz krb5kdc[5843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481765, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.125@KRBTEST.COM for kudu/127.15.185.125@KRBTEST.COM
I20260927 04:02:45.202109 6002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:45.194404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.125:45391 (local address 127.0.0.1:44108)
0927 04:02:45.194454 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:45.194465 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:45.194501 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:45.194813 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:45.194814 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:45.194964 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:02:45.195013 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:45.195015 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:45.196116 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:45.196117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:45.196639 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:45.196641 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:45.197164 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:45.197165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:45.197230 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:45.197694 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:02:45.197700 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:02:45.199082 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:02:45.200975 (+ 1893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:45.200976 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:45.200977 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:45.201159 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:45.201388 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:45.201389 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:45.201390 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:45.201446 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:45.201765 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:02:45.201766 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:02:45.201804 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 04:02:45.201877 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 04:02:45.201990 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 04:02:45.202133 6004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:45.194487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44108 (local address 127.15.185.125:45391)
0927 04:02:45.194675 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:45.194679 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:45.194699 (+ 20us) server_negotiation.cc:407] Connection header received
0927 04:02:45.194716 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:45.194718 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:45.194729 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:45.194813 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:45.195136 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:45.196010 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:45.196739 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:45.197066 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:45.197149 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:45.199193 (+ 2044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:45.199204 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:45.199205 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:45.199212 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:45.200860 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:45.201257 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:45.201261 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:45.201261 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:45.201319 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:45.201525 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:45.201525 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:45.201526 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:45.201693 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:45.201792 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:45.201920 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 04:02:45.202041 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":161,"thread_start_us":107,"threads_started":1}
I20260927 04:02:45.202993 5999 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.185.124" port: 42753 } has no permanent_uuid. Determining permanent_uuid...
I20260927 04:02:45.203471 6002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:45.203288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.124:42753 (local address 127.0.0.1:35752)
0927 04:02:45.203399 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:45.203447 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.124:42753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260927 04:02:45.203615 5947 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.185.124:42753: connect: Connection refused (error 111)
I20260927 04:02:45.204035 6002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:45.203932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.124:42753 (local address 127.0.0.1:35760)
0927 04:02:45.203987 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:45.204012 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.124:42753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260927 04:02:45.204203 5999 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.185.124:42753: Network error: Client connection negotiation failed: client connection to 127.15.185.124:42753: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.124.keytab.
Entry for principal kudu/127.15.185.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.124.keytab.
I20260927 04:02:45.211863 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.124:42753
--webserver_interface=127.15.185.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.124.keytab
--principal=kudu/127.15.185.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:40951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.185.126:45777,127.15.185.125:45391,127.15.185.124:42753
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 04:02:45.246356 5924 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.185.125" port: 45391 } attempt: 3
Sep 27 04:02:45 dist-test-slave-glgz krb5kdc[5843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481764, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.126@KRBTEST.COM for kudu/127.15.185.125@KRBTEST.COM
I20260927 04:02:45.253448 5999 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.185.124" port: 42753 } attempt: 1
I20260927 04:02:45.253756 5926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:45.246760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.125:45391 (local address 127.0.0.1:44114)
0927 04:02:45.246828 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:45.246841 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:45.246919 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:45.247097 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:45.247098 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:45.247271 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:02:45.247339 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:45.247341 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:45.248047 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:45.248053 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:45.248594 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:45.248596 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:45.249016 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:45.249017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:45.249108 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:45.249596 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:02:45.249600 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:02:45.250856 (+ 1256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:02:45.252613 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:45.252615 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:45.252615 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:45.252799 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:45.253022 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:45.253023 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:45.253023 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:45.253090 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:45.253396 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:02:45.253397 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:02:45.253447 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 04:02:45.253540 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 04:02:45.253651 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 04:02:45.253780 6004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:45.246849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44114 (local address 127.15.185.125:45391)
0927 04:02:45.246920 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:45.246922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:45.246933 (+ 11us) server_negotiation.cc:407] Connection header received
0927 04:02:45.246985 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:45.246986 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:45.247005 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:45.247089 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:45.247437 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:45.247945 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:45.248690 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:45.248924 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:45.249015 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:45.250955 (+ 1940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:45.250957 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:45.250957 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:45.250965 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:45.252503 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:45.252888 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:45.252889 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:45.252890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:45.252951 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:45.253173 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:45.253174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:45.253174 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:45.253298 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:45.253403 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:45.253565 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 04:02:45.253658 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 04:02:45.253899 6002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:45.253795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.124:42753 (local address 127.0.0.1:35770)
0927 04:02:45.253838 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:45.253876 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.124:42753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260927 04:02:45.254210 6002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:45.254114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.124:42753 (local address 127.0.0.1:35772)
0927 04:02:45.254164 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:45.254189 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.124:42753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 04:02:45.254391 5999 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.185.124:42753: Network error: Client connection negotiation failed: client connection to 127.15.185.124:42753: connect: Connection refused (error 111)
I20260927 04:02:45.254612 5924 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.185.124" port: 42753 } has no permanent_uuid. Determining permanent_uuid...
I20260927 04:02:45.254937 5926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:45.254839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.124:42753 (local address 127.0.0.1:35776)
0927 04:02:45.254887 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:45.254913 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.124:42753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 04:02:45.255296 5926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:45.255208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.124:42753 (local address 127.0.0.1:35782)
0927 04:02:45.255253 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:45.255275 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.124:42753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 04:02:45.255396 5924 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.185.124:42753: Network error: Client connection negotiation failed: client connection to 127.15.185.124:42753: connect: Connection refused (error 111)
I20260927 04:02:45.286527 5924 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.185.124" port: 42753 } attempt: 1
I20260927 04:02:45.287009 5926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:45.286869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.124:42753 (local address 127.0.0.1:35788)
0927 04:02:45.286939 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:45.286978 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.124:42753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 04:02:45.287431 5926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:45.287332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.124:42753 (local address 127.0.0.1:35790)
0927 04:02:45.287380 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:45.287407 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.124:42753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260927 04:02:45.287523 5924 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.185.124:42753: Network error: Client connection negotiation failed: client connection to 127.15.185.124:42753: connect: Connection refused (error 111)
W20260927 04:02:45.297115 6006 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 04:02:45.297360 6006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:02:45.297391 6006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:02:45.297413 6006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:02:45.298692 6006 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:02:45.298738 6006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:02:45.298753 6006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:02:45.298764 6006 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:02:45.298776 6006 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:02:45.300340 6006 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:40951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.15.185.126:45777,127.15.185.125:45391,127.15.185.124:42753
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.124.keytab
--principal=kudu/127.15.185.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.124:42753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.15.185.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:02:45.300621 6006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:02:45.300864 6006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:02:45.306172 6011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:45.306218 6014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:45.306373 6006 server_base.cc:1034] running on GCE node
W20260927 04:02:45.306218 6012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:45.306783 6006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:02:45.307111 6006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:02:45.308300 6006 hybrid_clock.cc:648] HybridClock initialized: now 1790481765308284 us; error 31 us; skew 500 ppm
Sep 27 04:02:45 dist-test-slave-glgz krb5kdc[5843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481765, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:02:45.311230 6006 init.cc:377] Logged in from keytab as kudu/127.15.185.124@KRBTEST.COM (short username kudu)
I20260927 04:02:45.311684 6006 webserver.cc:460] Webserver started at http://127.15.185.124:35351/ using document root <none> and password file <none>
I20260927 04:02:45.311928 6006 fs_manager.cc:359] Metadata directory not provided
I20260927 04:02:45.311998 6006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:02:45.312155 6006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:02:45.313289 6006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/minicluster-data/master-2/data/instance:
uuid: "3d805ac0e6fa4c7986d14d03cf2d4033"
format_stamp: "Formatted at 2026-09-27 04:02:45 on dist-test-slave-glgz"
server_key: "f4d6418d377a8561ad67af00ccd807cb"
server_key_iv: "34ed2b9add20ef659463df9f4c4555fd"
server_key_version: "encryptionkey@0"
I20260927 04:02:45.313673 6006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/minicluster-data/master-2/wal/instance:
uuid: "3d805ac0e6fa4c7986d14d03cf2d4033"
format_stamp: "Formatted at 2026-09-27 04:02:45 on dist-test-slave-glgz"
server_key: "f4d6418d377a8561ad67af00ccd807cb"
server_key_iv: "34ed2b9add20ef659463df9f4c4555fd"
server_key_version: "encryptionkey@0"
I20260927 04:02:45.315245 6006 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 04:02:45.316191 6021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:45.316442 6006 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:02:45.316556 6006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/minicluster-data/master-2/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/minicluster-data/master-2/wal
uuid: "3d805ac0e6fa4c7986d14d03cf2d4033"
format_stamp: "Formatted at 2026-09-27 04:02:45 on dist-test-slave-glgz"
server_key: "f4d6418d377a8561ad67af00ccd807cb"
server_key_iv: "34ed2b9add20ef659463df9f4c4555fd"
server_key_version: "encryptionkey@0"
I20260927 04:02:45.316641 6006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:02:45.322562 5999 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.185.124" port: 42753 } attempt: 2
I20260927 04:02:45.323086 6002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:45.322918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.124:42753 (local address 127.0.0.1:35792)
0927 04:02:45.322978 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:45.323022 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.124:42753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 04:02:45.323447 6002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:45.323348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.124:42753 (local address 127.0.0.1:35806)
0927 04:02:45.323395 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:45.323424 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.124:42753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 04:02:45.323577 5999 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.185.124:42753: Network error: Client connection negotiation failed: client connection to 127.15.185.124:42753: connect: Connection refused (error 111)
I20260927 04:02:45.332193 6006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:02:45.334743 6006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:02:45.334918 6006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:02:45.338892 6006 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.124:42753
I20260927 04:02:45.338950 6073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.124:42753 every 8 connection(s)
I20260927 04:02:45.339445 6006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/minicluster-data/master-2/data/info.pb
I20260927 04:02:45.340462 6074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:02:45.342070 6074 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45777 } has no permanent_uuid. Determining permanent_uuid...
I20260927 04:02:45.347270 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 6006
I20260927 04:02:45.347386 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481498018407-16101-0/minicluster-data/master-2/wal/instance
I20260927 04:02:45.347613 16101 external_mini_cluster.cc:1428] Setting key defc6ba71d50af4b874d852ae6f22de1
Sep 27 04:02:45 dist-test-slave-glgz krb5kdc[5843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481765, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.124@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 04:02:45.351720 5927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:45.342856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43352 (local address 127.15.185.126:45777)
0927 04:02:45.342938 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:45.342940 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:45.343168 (+ 228us) server_negotiation.cc:407] Connection header received
0927 04:02:45.343618 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:45.343619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:45.343629 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:45.343684 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:45.344429 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:45.344909 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:45.345995 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:45.346202 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:45.346262 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:45.348677 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:45.348679 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:45.348679 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:45.348695 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:45.350176 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:45.350626 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:45.350627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:45.350628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:45.350694 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:45.350969 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:45.350969 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:45.350970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:45.351173 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:45.351248 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:45.351530 (+ 282us) server_negotiation.cc:299] Negotiation successful
0927 04:02:45.351613 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 04:02:45.351922 6076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:45.342772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:45777 (local address 127.0.0.1:43352)
0927 04:02:45.342978 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:45.343015 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:45.343534 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:45.343852 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:45.343856 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:45.344132 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:02:45.344342 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:45.344347 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:45.344996 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:45.344997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:45.345905 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:45.345908 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:45.346281 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:45.346282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:45.346390 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:45.346888 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:02:45.346897 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:02:45.348552 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:02:45.350282 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:45.350290 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:45.350292 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:45.350533 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:45.350765 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:45.350765 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:45.350766 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:45.350889 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:45.351239 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:02:45.351246 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:02:45.351395 (+ 149us) client_negotiation.cc:769] Sending connection context
0927 04:02:45.351508 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 04:02:45.351671 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":109,"threads_started":1}
I20260927 04:02:45.352659 6074 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.185.125" port: 45391 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 04:02:45 dist-test-slave-glgz krb5kdc[5843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.185.124@KRBTEST.COM
Sep 27 04:02:45 dist-test-slave-glgz krb5kdc[5843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481765, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.124@KRBTEST.COM for kudu/127.15.185.125@KRBTEST.COM
I20260927 04:02:45.357916 6078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:45.348153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35820 (local address 127.15.185.124:42753)
0927 04:02:45.348499 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:45.348503 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:45.348520 (+ 17us) server_negotiation.cc:407] Connection header received
0927 04:02:45.348539 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:45.348541 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:45.348551 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:45.348642 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:45.349340 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:45.350050 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:45.350839 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:45.351268 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:45.351346 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:45.354040 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:45.354048 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:45.354048 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:45.354055 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:45.356003 (+ 1948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:45.356896 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:45.356900 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:45.356900 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:45.356955 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:45.357206 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:45.357207 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:45.357207 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:45.357350 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:45.357536 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:45.357777 (+ 241us) server_negotiation.cc:299] Negotiation successful
0927 04:02:45.357843 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":102,"threads_started":1}
I20260927 04:02:45.360023 6076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:45.352975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.125:45391 (local address 127.0.0.1:44124)
0927 04:02:45.353102 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:45.353113 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:45.353165 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:45.353284 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:45.353285 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:45.353429 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:02:45.353511 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:45.353512 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:45.354196 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:45.354197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:45.355023 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:45.355025 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:45.355458 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:45.355459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:45.355529 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:45.355838 (+ 309us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:02:45.355843 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:02:45.357086 (+ 1243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:02:45.358953 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:45.358953 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:45.358954 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:45.359194 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:45.359418 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:45.359419 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:45.359419 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:45.359464 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:45.359736 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:02:45.359736 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:02:45.359766 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 04:02:45.359843 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 04:02:45.359929 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110}
I20260927 04:02:45.360039 6004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:45.353051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44124 (local address 127.15.185.125:45391)
0927 04:02:45.353120 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:45.353122 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:45.353150 (+ 28us) server_negotiation.cc:407] Connection header received
0927 04:02:45.353223 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:45.353224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:45.353233 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:45.353281 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:45.353579 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:45.354119 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:45.355185 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:45.355377 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:45.355439 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:45.357175 (+ 1736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:45.357177 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:45.357177 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:45.357185 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:45.358840 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:45.359273 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:45.359275 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:45.359275 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:45.359342 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:45.359505 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:45.359506 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:45.359506 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:45.359668 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:45.359750 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:45.359892 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 04:02:45.359965 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 04:02:45.360599 6074 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.185.124" port: 42753 } has no permanent_uuid. Determining permanent_uuid...
I20260927 04:02:45.362675 5924 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.185.124" port: 42753 } attempt: 2
Sep 27 04:02:45 dist-test-slave-glgz krb5kdc[5843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
Sep 27 04:02:45 dist-test-slave-glgz krb5kdc[5843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481765, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.124@KRBTEST.COM for kudu/127.15.185.124@KRBTEST.COM
I20260927 04:02:45.367654 5927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:45.358951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43356 (local address 127.15.185.126:45777)
0927 04:02:45.358994 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:45.358996 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:45.359003 (+ 7us) server_negotiation.cc:407] Connection header received
0927 04:02:45.359033 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:45.359034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:45.359089 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:45.359148 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:45.359824 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:45.360551 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:45.361402 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:45.361695 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:45.361763 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:45.364620 (+ 2857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:45.364621 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:45.364621 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:45.364629 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:45.366345 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:45.366763 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:45.366764 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:45.366764 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:45.366815 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:45.367042 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:45.367079 (+ 37us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:45.367080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:45.367197 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:45.367409 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:45.367447 (+ 38us) server_negotiation.cc:299] Negotiation successful
0927 04:02:45.367523 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Sep 27 04:02:45 dist-test-slave-glgz krb5kdc[5843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481764, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.126@KRBTEST.COM for kudu/127.15.185.124@KRBTEST.COM
I20260927 04:02:45.368779 6076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:45.360860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.124:42753 (local address 127.0.0.1:35830)
0927 04:02:45.360892 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:45.360902 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:45.360937 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:45.361038 (+ 101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:45.361038 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:45.361177 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:02:45.361204 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:45.361205 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:45.362056 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:45.362056 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:45.363022 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:45.363026 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:45.363450 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:45.363451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:45.363499 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:45.363931 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:02:45.363937 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:02:45.365407 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:02:45.367604 (+ 2197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:45.367604 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:45.367605 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:45.367817 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:45.368004 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:45.368005 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:45.368005 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:45.368061 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:45.368423 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:02:45.368424 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:02:45.368467 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 04:02:45.368538 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 04:02:45.368647 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20260927 04:02:45.368882 6078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:45.360915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35830 (local address 127.15.185.124:42753)
0927 04:02:45.360948 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:45.360950 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:45.360958 (+ 8us) server_negotiation.cc:407] Connection header received
0927 04:02:45.360985 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:45.360986 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:45.360995 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:45.361037 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:45.361282 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:45.361979 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:45.363157 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:45.363369 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:45.363442 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:45.365498 (+ 2056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:45.365499 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:45.365500 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:45.365506 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:45.367516 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:45.367899 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:45.367900 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:45.367900 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:45.367947 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:45.368134 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:45.368135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:45.368135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:45.368278 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:45.368402 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:45.368572 (+ 170us) server_negotiation.cc:299] Negotiation successful
0927 04:02:45.368752 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":18}
I20260927 04:02:45.371308 6074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d805ac0e6fa4c7986d14d03cf2d4033: Bootstrap starting.
I20260927 04:02:45.372071 6074 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d805ac0e6fa4c7986d14d03cf2d4033: Neither blocks nor log segments found. Creating new log.
I20260927 04:02:45.372396 6074 log.cc:824] T 00000000000000000000000000000000 P 3d805ac0e6fa4c7986d14d03cf2d4033: Log is configured to *not* fsync() on all Append() calls
Sep 27 04:02:45 dist-test-slave-glgz krb5kdc[5843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.185.125@KRBTEST.COM
I20260927 04:02:45.373661 6074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d805ac0e6fa4c7986d14d03cf2d4033: No bootstrap required, opened a new log
I20260927 04:02:45.375505 5926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:45.362938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.124:42753 (local address 127.0.0.1:35844)
0927 04:02:45.362985 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:45.362994 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:45.363031 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:45.363551 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:45.363552 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:45.363712 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:02:45.363744 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:45.363745 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:45.364661 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:45.364662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:45.365569 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:45.365572 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:45.366072 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:45.366073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:45.366141 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:45.366810 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:02:45.366817 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:02:45.368701 (+ 1884us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:02:45.374059 (+ 5358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:45.374061 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:45.374061 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:45.374324 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:45.374660 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:45.374661 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:45.374661 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:45.374762 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:45.375163 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:02:45.375164 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:02:45.375215 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 04:02:45.375290 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 04:02:45.375401 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 04:02:45.375627 6079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:45.362967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35844 (local address 127.15.185.124:42753)
0927 04:02:45.363419 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:45.363421 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:45.363429 (+ 8us) server_negotiation.cc:407] Connection header received
0927 04:02:45.363452 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:45.363453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:45.363465 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:45.363544 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:45.363845 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:45.364588 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:45.365669 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:45.365988 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:45.366055 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:45.368808 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:45.368810 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:45.368811 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:45.368817 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:45.373950 (+ 5133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:45.374434 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:45.374435 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:45.374435 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:45.374561 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:45.374851 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:45.374853 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:45.374853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:45.375082 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:45.375159 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:45.375345 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 04:02:45.375441 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":137,"threads_started":1}
I20260927 04:02:45.376278 6074 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d805ac0e6fa4c7986d14d03cf2d4033 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0117710d7e544188d814baa51cf1230" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45777 } } peers { permanent_uuid: "6a62c49c62384f649489cb5dc8d9dda8" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 45391 } } peers { permanent_uuid: "3d805ac0e6fa4c7986d14d03cf2d4033" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 42753 } }
I20260927 04:02:45.376531 6074 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d805ac0e6fa4c7986d14d03cf2d4033 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d805ac0e6fa4c7986d14d03cf2d4033, State: Initialized, Role: FOLLOWER
I20260927 04:02:45.376652 6074 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d805ac0e6fa4c7986d14d03cf2d4033 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0117710d7e544188d814baa51cf1230" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45777 } } peers { permanent_uuid: "6a62c49c62384f649489cb5dc8d9dda8" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 45391 } } peers { permanent_uuid: "3d805ac0e6fa4c7986d14d03cf2d4033" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 42753 } }
I20260927 04:02:45.377019 6074 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d805ac0e6fa4c7986d14d03cf2d4033 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:02:45.376967 6081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d805ac0e6fa4c7986d14d03cf2d4033 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0117710d7e544188d814baa51cf1230" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45777 } } peers { permanent_uuid: "6a62c49c62384f649489cb5dc8d9dda8" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 45391 } } peers { permanent_uuid: "3d805ac0e6fa4c7986d14d03cf2d4033" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 42753 } } }
I20260927 04:02:45.377225 6081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d805ac0e6fa4c7986d14d03cf2d4033 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 04:02:45.377801 6004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:45.368857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44140 (local address 127.15.185.125:45391)
0927 04:02:45.368915 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:45.368917 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:45.368948 (+ 31us) server_negotiation.cc:407] Connection header received
0927 04:02:45.369021 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:45.369022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:45.369032 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:45.369078 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:45.369705 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:45.370454 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:45.371244 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:45.371541 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:45.371625 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:45.374177 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:45.374179 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:45.374179 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:45.374189 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:45.376066 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:45.376505 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:45.376507 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:45.376507 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:45.376575 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:45.376814 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:45.376815 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:45.376815 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:45.376946 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:45.377112 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:45.377563 (+ 451us) server_negotiation.cc:299] Negotiation successful
0927 04:02:45.377666 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 04:02:45.378334 5924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0117710d7e544188d814baa51cf1230: Bootstrap starting.
I20260927 04:02:45.379165 5924 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0117710d7e544188d814baa51cf1230: Neither blocks nor log segments found. Creating new log.
W20260927 04:02:45.379206 6092 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3d805ac0e6fa4c7986d14d03cf2d4033: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 04:02:45.379249 6092 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 04:02:45.379498 5924 log.cc:824] T 00000000000000000000000000000000 P d0117710d7e544188d814baa51cf1230: Log is configured to *not* fsync() on all Append() calls
I20260927 04:02:45.379958 16101 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 04:02:45.380391 5924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0117710d7e544188d814baa51cf1230: No bootstrap required, opened a new log
I20260927 04:02:45.380945 5898 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d0117710d7e544188d814baa51cf1230"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43356
I20260927 04:02:45.381076 6006 master_runner.cc:186] Error getting master registration for 127.15.185.126:45777: OK, instance_id { permanent_uuid: "d0117710d7e544188d814baa51cf1230" instance_seqno: 1790481765017958 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 04:02:45.382952 6006 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 04:02:45.383307 5924 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0117710d7e544188d814baa51cf1230 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0117710d7e544188d814baa51cf1230" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45777 } } peers { permanent_uuid: "6a62c49c62384f649489cb5dc8d9dda8" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 45391 } } peers { permanent_uuid: "3d805ac0e6fa4c7986d14d03cf2d4033" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 42753 } }
I20260927 04:02:45.383461 5924 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0117710d7e544188d814baa51cf1230 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0117710d7e544188d814baa51cf1230, State: Initialized, Role: FOLLOWER
I20260927 04:02:45.383543 5924 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0117710d7e544188d814baa51cf1230 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0117710d7e544188d814baa51cf1230" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45777 } } peers { permanent_uuid: "6a62c49c62384f649489cb5dc8d9dda8" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 45391 } } peers { permanent_uuid: "3d805ac0e6fa4c7986d14d03cf2d4033" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 42753 } }
I20260927 04:02:45.383816 6095 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0117710d7e544188d814baa51cf1230 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0117710d7e544188d814baa51cf1230" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45777 } } peers { permanent_uuid: "6a62c49c62384f649489cb5dc8d9dda8" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 45391 } } peers { permanent_uuid: "3d805ac0e6fa4c7986d14d03cf2d4033" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 42753 } } }
I20260927 04:02:45.383877 5924 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0117710d7e544188d814baa51cf1230 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:02:45.383893 5898 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0117710d7e544188d814baa51cf1230 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 04:02:45.384019 5898 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0117710d7e544188d814baa51cf1230 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:02:45.384879 5898 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0117710d7e544188d814baa51cf1230 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0117710d7e544188d814baa51cf1230" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45777 } } peers { permanent_uuid: "6a62c49c62384f649489cb5dc8d9dda8" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 45391 } } peers { permanent_uuid: "3d805ac0e6fa4c7986d14d03cf2d4033" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 42753 } }
I20260927 04:02:45.385185 5898 leader_election.cc:288] T 00000000000000000000000000000000 P d0117710d7e544188d814baa51cf1230 [CANDIDATE]: Term 1 election: Requested vote from peers 6a62c49c62384f649489cb5dc8d9dda8 (127.15.185.125:45391), 3d805ac0e6fa4c7986d14d03cf2d4033 (127.15.185.124:42753)
W20260927 04:02:45.385908 6106 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d0117710d7e544188d814baa51cf1230: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 04:02:45.386027 6106 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 04:02:45.386269 6048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d0117710d7e544188d814baa51cf1230" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3d805ac0e6fa4c7986d14d03cf2d4033"
I20260927 04:02:45.386400 6048 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d805ac0e6fa4c7986d14d03cf2d4033 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:02:45.387195 6048 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3d805ac0e6fa4c7986d14d03cf2d4033 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0117710d7e544188d814baa51cf1230 in term 1.
I20260927 04:02:45.387641 5872 leader_election.cc:302] T 00000000000000000000000000000000 P d0117710d7e544188d814baa51cf1230 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d805ac0e6fa4c7986d14d03cf2d4033, d0117710d7e544188d814baa51cf1230; no voters:
I20260927 04:02:45.383944 6095 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0117710d7e544188d814baa51cf1230 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 04:02:45.388768 6095 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0117710d7e544188d814baa51cf1230 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:02:45.388849 6095 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0117710d7e544188d814baa51cf1230 [term 1 LEADER]: Becoming Leader. State: Replica: d0117710d7e544188d814baa51cf1230, State: Running, Role: LEADER
I20260927 04:02:45.388976 6095 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0117710d7e544188d814baa51cf1230 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0117710d7e544188d814baa51cf1230" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45777 } } peers { permanent_uuid: "6a62c49c62384f649489cb5dc8d9dda8" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 45391 } } peers { permanent_uuid: "3d805ac0e6fa4c7986d14d03cf2d4033" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 42753 } }
I20260927 04:02:45.389688 6095 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0117710d7e544188d814baa51cf1230 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0117710d7e544188d814baa51cf1230. Latest consensus state: current_term: 1 leader_uuid: "d0117710d7e544188d814baa51cf1230" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0117710d7e544188d814baa51cf1230" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45777 } } peers { permanent_uuid: "6a62c49c62384f649489cb5dc8d9dda8" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 45391 } } peers { permanent_uuid: "3d805ac0e6fa4c7986d14d03cf2d4033" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 42753 } } }
I20260927 04:02:45.389819 6095 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0117710d7e544188d814baa51cf1230 [sys.catalog]: This master's current role is: LEADER
I20260927 04:02:45.387647 5856 master_runner.cc:186] Error getting master registration for 127.15.185.125:45391: OK, instance_id { permanent_uuid: "6a62c49c62384f649489cb5dc8d9dda8" instance_seqno: 1790481765178428 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 04:02:45.390208 5856 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 04:02:45.396701 6114 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:02:45.397003 6114 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:02:45.399003 6048 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3d805ac0e6fa4c7986d14d03cf2d4033 [term 1 FOLLOWER]: Refusing update from remote peer d0117710d7e544188d814baa51cf1230: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:02:45.399446 6108 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d0117710d7e544188d814baa51cf1230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d805ac0e6fa4c7986d14d03cf2d4033" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 42753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:02:45.401916 6081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d805ac0e6fa4c7986d14d03cf2d4033 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0117710d7e544188d814baa51cf1230. Latest consensus state: current_term: 1 leader_uuid: "d0117710d7e544188d814baa51cf1230" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0117710d7e544188d814baa51cf1230" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45777 } } peers { permanent_uuid: "6a62c49c62384f649489cb5dc8d9dda8" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 45391 } } peers { permanent_uuid: "3d805ac0e6fa4c7986d14d03cf2d4033" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 42753 } } }
I20260927 04:02:45.402040 6081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d805ac0e6fa4c7986d14d03cf2d4033 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 04:02:45.408380 6108 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0117710d7e544188d814baa51cf1230 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d0117710d7e544188d814baa51cf1230" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0117710d7e544188d814baa51cf1230" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45777 } } peers { permanent_uuid: "6a62c49c62384f649489cb5dc8d9dda8" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 45391 } } peers { permanent_uuid: "3d805ac0e6fa4c7986d14d03cf2d4033" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 42753 } } }
I20260927 04:02:45.408506 6108 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0117710d7e544188d814baa51cf1230 [sys.catalog]: This master's current role is: LEADER
I20260927 04:02:45.409104 6081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d805ac0e6fa4c7986d14d03cf2d4033 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d0117710d7e544188d814baa51cf1230" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0117710d7e544188d814baa51cf1230" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45777 } } peers { permanent_uuid: "6a62c49c62384f649489cb5dc8d9dda8" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 45391 } } peers { permanent_uuid: "3d805ac0e6fa4c7986d14d03cf2d4033" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 42753 } } }
I20260927 04:02:45.409251 6081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d805ac0e6fa4c7986d14d03cf2d4033 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 04:02:45.409327 6119 mvcc.cc:205] Tried to move back new op lower bound from 7333813311071039488 to 7333813311034634240. Current Snapshot: MvccSnapshot[applied={T|T < 7333813311071039488}]
I20260927 04:02:45.409768 6114 catalog_manager.cc:1347] Generated new cluster ID: 84faabecb13c413d8512aba2d1713625
I20260927 04:02:45.409825 6114 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:02:45.413882 6079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:45.395425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35848 (local address 127.15.185.124:42753)
0927 04:02:45.395499 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:45.395502 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:45.396305 (+ 803us) server_negotiation.cc:407] Connection header received
0927 04:02:45.396556 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:45.396558 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:45.396585 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:45.396622 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:45.401036 (+ 4414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:45.401795 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:45.404210 (+ 2415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:45.404556 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:45.404624 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:45.410063 (+ 5439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:45.410065 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:45.410065 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:45.410073 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:45.411736 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:45.412390 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:45.412391 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:45.412392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:45.412448 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:45.412870 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:45.412871 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:45.412871 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:45.412997 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:45.413081 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:45.413676 (+ 595us) server_negotiation.cc:299] Negotiation successful
0927 04:02:45.413782 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 04:02:45.414709 5999 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.185.124" port: 42753 } attempt: 3
Sep 27 04:02:45 dist-test-slave-glgz krb5kdc[5843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481765, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.125@KRBTEST.COM for kudu/127.15.185.124@KRBTEST.COM
I20260927 04:02:45.420883 6004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:45.395940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44156 (local address 127.15.185.125:45391)
0927 04:02:45.395986 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:45.395988 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:45.398371 (+ 2383us) server_negotiation.cc:407] Connection header received
0927 04:02:45.411160 (+ 12789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:45.411163 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:45.411181 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:45.411218 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:45.411949 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:45.412742 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:45.413876 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:45.414218 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:45.414262 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:45.415855 (+ 1593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:45.415857 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:45.415857 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:45.415865 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:45.417564 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:45.417984 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:45.417985 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:45.417986 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:45.418044 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:45.418523 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:45.418524 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:45.418525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:45.418668 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:45.418777 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:45.420677 (+ 1900us) server_negotiation.cc:299] Negotiation successful
0927 04:02:45.420773 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 04:02:45.424635 6002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:45.414959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.124:42753 (local address 127.0.0.1:35852)
0927 04:02:45.415065 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:45.415076 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:45.415114 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:45.415371 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:45.415372 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:45.415551 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:02:45.415594 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:45.415596 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:45.416543 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:45.416544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:45.417356 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:45.417359 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:45.417901 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:45.417902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:45.417960 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:45.419289 (+ 1329us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:02:45.419294 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:02:45.421619 (+ 2325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:02:45.423552 (+ 1933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:45.423554 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:45.423554 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:45.423767 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:45.423968 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:45.423969 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:45.423969 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:45.424025 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:45.424310 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:02:45.424311 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:02:45.424360 (+ 49us) client_negotiation.cc:769] Sending connection context
0927 04:02:45.424422 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 04:02:45.424521 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 04:02:45.424997 6079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:45.415207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35852 (local address 127.15.185.124:42753)
0927 04:02:45.415245 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:45.415246 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:45.415253 (+ 7us) server_negotiation.cc:407] Connection header received
0927 04:02:45.415274 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:45.415275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:45.415284 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:45.415330 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:45.415676 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:45.416440 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:45.417477 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:45.417779 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:45.417841 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:45.421725 (+ 3884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:45.421727 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:45.421728 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:45.421736 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:45.423450 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:45.423843 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:45.423844 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:45.423844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:45.423901 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:45.424088 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:45.424089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:45.424089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:45.424231 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:45.424287 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:45.424833 (+ 546us) server_negotiation.cc:299] Negotiation successful
0927 04:02:45.424912 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 04:02:45.425539 5927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:45.395810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43362 (local address 127.15.185.126:45777)
0927 04:02:45.395898 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:45.395901 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:45.396219 (+ 318us) server_negotiation.cc:407] Connection header received
0927 04:02:45.396732 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:45.396733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:45.396744 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:45.396774 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:45.402114 (+ 5340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:45.402820 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:45.404922 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:45.405217 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:45.405281 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:45.406984 (+ 1703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:45.406986 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:45.406987 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:45.406997 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:45.408725 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:45.412628 (+ 3903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:45.412629 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:45.412629 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:45.412689 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:45.413260 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:45.413261 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:45.413261 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:45.413390 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:45.413464 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:45.425339 (+ 11875us) server_negotiation.cc:299] Negotiation successful
0927 04:02:45.425437 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 04:02:45.426826 5999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a62c49c62384f649489cb5dc8d9dda8: Bootstrap starting.
I20260927 04:02:45.427722 5999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a62c49c62384f649489cb5dc8d9dda8: Neither blocks nor log segments found. Creating new log.
I20260927 04:02:45.428052 5999 log.cc:824] T 00000000000000000000000000000000 P 6a62c49c62384f649489cb5dc8d9dda8: Log is configured to *not* fsync() on all Append() calls
I20260927 04:02:45.438856 5999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a62c49c62384f649489cb5dc8d9dda8: No bootstrap required, opened a new log
I20260927 04:02:45.438944 6114 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:02:45.440001 6114 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:02:45.441430 5999 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a62c49c62384f649489cb5dc8d9dda8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0117710d7e544188d814baa51cf1230" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45777 } } peers { permanent_uuid: "6a62c49c62384f649489cb5dc8d9dda8" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 45391 } } peers { permanent_uuid: "3d805ac0e6fa4c7986d14d03cf2d4033" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 42753 } }
I20260927 04:02:45.441610 5999 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a62c49c62384f649489cb5dc8d9dda8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a62c49c62384f649489cb5dc8d9dda8, State: Initialized, Role: FOLLOWER
I20260927 04:02:45.441713 5999 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a62c49c62384f649489cb5dc8d9dda8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0117710d7e544188d814baa51cf1230" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45777 } } peers { permanent_uuid: "6a62c49c62384f649489cb5dc8d9dda8" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 45391 } } peers { permanent_uuid: "3d805ac0e6fa4c7986d14d03cf2d4033" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 42753 } }
I20260927 04:02:45.442098 5999 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a62c49c62384f649489cb5dc8d9dda8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:02:45.442196 5972 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a62c49c62384f649489cb5dc8d9dda8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:02:45.444075 5972 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6a62c49c62384f649489cb5dc8d9dda8 [term 1 FOLLOWER]: Refusing update from remote peer d0117710d7e544188d814baa51cf1230: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:02:45.442090 6124 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a62c49c62384f649489cb5dc8d9dda8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0117710d7e544188d814baa51cf1230" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45777 } } peers { permanent_uuid: "6a62c49c62384f649489cb5dc8d9dda8" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 45391 } } peers { permanent_uuid: "3d805ac0e6fa4c7986d14d03cf2d4033" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 42753 } } }
I20260927 04:02:45.444514 6108 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d0117710d7e544188d814baa51cf1230 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a62c49c62384f649489cb5dc8d9dda8" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 45391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:02:45.442219 5973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d0117710d7e544188d814baa51cf1230" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6a62c49c62384f649489cb5dc8d9dda8"
I20260927 04:02:45.444676 6124 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a62c49c62384f649489cb5dc8d9dda8 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 04:02:45.445425 5973 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6a62c49c62384f649489cb5dc8d9dda8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0117710d7e544188d814baa51cf1230 in term 1.
W20260927 04:02:45.446323 6135 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6a62c49c62384f649489cb5dc8d9dda8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 04:02:45.446431 6135 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 04:02:45.449189 6124 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a62c49c62384f649489cb5dc8d9dda8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0117710d7e544188d814baa51cf1230. Latest consensus state: current_term: 1 leader_uuid: "d0117710d7e544188d814baa51cf1230" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0117710d7e544188d814baa51cf1230" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45777 } } peers { permanent_uuid: "6a62c49c62384f649489cb5dc8d9dda8" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 45391 } } peers { permanent_uuid: "3d805ac0e6fa4c7986d14d03cf2d4033" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 42753 } } }
I20260927 04:02:45.449290 6124 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a62c49c62384f649489cb5dc8d9dda8 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 04:02:45.449324 6124 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a62c49c62384f649489cb5dc8d9dda8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d0117710d7e544188d814baa51cf1230" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0117710d7e544188d814baa51cf1230" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45777 } } peers { permanent_uuid: "6a62c49c62384f649489cb5dc8d9dda8" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 45391 } } peers { permanent_uuid: "3d805ac0e6fa4c7986d14d03cf2d4033" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 42753 } } }
I20260927 04:02:45.449366 6124 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a62c49c62384f649489cb5dc8d9dda8 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 04:02:45.451710 6137 mvcc.cc:205] Tried to move back new op lower bound from 7333813311233003520 to 7333813311034634240. Current Snapshot: MvccSnapshot[applied={T|T < 7333813311071039488}]
I20260927 04:02:45.457017 6114 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0117710d7e544188d814baa51cf1230: Generated new TSK 0
I20260927 04:02:45.457235 6114 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 04:02:45.458093 6095 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0117710d7e544188d814baa51cf1230 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d0117710d7e544188d814baa51cf1230" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0117710d7e544188d814baa51cf1230" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 45777 } } peers { permanent_uuid: "6a62c49c62384f649489cb5dc8d9dda8" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 45391 } } peers { permanent_uuid: "3d805ac0e6fa4c7986d14d03cf2d4033" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 42753 } } }
I20260927 04:02:45.458199 6095 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0117710d7e544188d814baa51cf1230 [sys.catalog]: This master's current role is: LEADER
W20260927 04:02:45.489593 5927 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:02:45.488303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43366 (local address 127.15.185.126:45777)
0927 04:02:45.488348 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:45.488350 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:45.488931 (+ 581us) server_negotiation.cc:407] Connection header received
0927 04:02:45.489035 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:45.489037 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:45.489048 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:45.489078 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:45.489515 (+ 437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43366: BlockingRecv error: recv got EOF from 127.0.0.1:43366 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260927 04:02:45.490633 6004 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:02:45.488667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44168 (local address 127.15.185.125:45391)
0927 04:02:45.488708 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:45.488709 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:45.490079 (+ 1370us) server_negotiation.cc:407] Connection header received
0927 04:02:45.490217 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:45.490218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:45.490229 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:45.490258 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:45.490572 (+ 314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44168: BlockingRecv error: recv got EOF from 127.0.0.1:44168 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260927 04:02:45.491513 6079 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:02:45.489961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35862 (local address 127.15.185.124:42753)
0927 04:02:45.490000 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:45.490002 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:45.490867 (+ 865us) server_negotiation.cc:407] Connection header received
0927 04:02:45.491030 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:45.491031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:45.491063 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:45.491095 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:45.491450 (+ 355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35862: BlockingRecv error: recv got EOF from 127.0.0.1:35862 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260927 04:02:45.493916 6004 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:02:45.493101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44176 (local address 127.15.185.125:45391)
0927 04:02:45.493137 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:45.493139 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:45.493357 (+ 218us) server_negotiation.cc:407] Connection header received
0927 04:02:45.493464 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:45.493465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:45.493487 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:45.493543 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:45.493867 (+ 324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44176: BlockingRecv error: recv got EOF from 127.0.0.1:44176 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260927 04:02:45.496953 6004 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:02:45.496142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44184 (local address 127.15.185.125:45391)
0927 04:02:45.496186 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:45.496188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:45.496445 (+ 257us) server_negotiation.cc:407] Connection header received
0927 04:02:45.496542 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:45.496543 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:45.496554 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:45.496583 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:45.496905 (+ 322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44184: BlockingRecv error: recv got EOF from 127.0.0.1:44184 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260927 04:02:45.500984 6004 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:02:45.500115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44194 (local address 127.15.185.125:45391)
0927 04:02:45.500154 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:45.500156 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:45.500421 (+ 265us) server_negotiation.cc:407] Connection header received
0927 04:02:45.500525 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:45.500526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:45.500537 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:45.500568 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:45.500930 (+ 362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44194: BlockingRecv error: recv got EOF from 127.0.0.1:44194 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260927 04:02:45.507230 6004 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:02:45.506251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44200 (local address 127.15.185.125:45391)
0927 04:02:45.506296 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:45.506298 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:45.506585 (+ 287us) server_negotiation.cc:407] Connection header received
0927 04:02:45.506709 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:45.506710 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:45.506722 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:45.506753 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:45.507170 (+ 417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44200: BlockingRecv error: recv got EOF from 127.0.0.1:44200 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260927 04:02:45.517658 6004 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:02:45.516611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44204 (local address 127.15.185.125:45391)
0927 04:02:45.516675 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:45.516677 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:45.516995 (+ 318us) server_negotiation.cc:407] Connection header received
0927 04:02:45.517103 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:45.517104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:45.517116 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:45.517147 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:45.517569 (+ 422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44204: BlockingRecv error: recv got EOF from 127.0.0.1:44204 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260927 04:02:45.536139 6004 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:02:45.535096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44216 (local address 127.15.185.125:45391)
0927 04:02:45.535145 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:45.535148 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:45.535451 (+ 303us) server_negotiation.cc:407] Connection header received
0927 04:02:45.535560 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:45.535562 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:45.535574 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:45.535604 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:45.536072 (+ 468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44216: BlockingRecv error: recv got EOF from 127.0.0.1:44216 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260927 04:02:45.570524 6004 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:02:45.569585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44232 (local address 127.15.185.125:45391)
0927 04:02:45.569629 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:45.569632 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:45.569919 (+ 287us) server_negotiation.cc:407] Connection header received
0927 04:02:45.570043 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:45.570044 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:45.570055 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:45.570085 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:45.570457 (+ 372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44232: BlockingRecv error: recv got EOF from 127.0.0.1:44232 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260927 04:02:45.637627 6004 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:02:45.636701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44236 (local address 127.15.185.125:45391)
0927 04:02:45.636788 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:45.636790 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:45.636816 (+ 26us) server_negotiation.cc:407] Connection header received
0927 04:02:45.636909 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:45.636910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:45.636921 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:45.636981 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:45.637523 (+ 542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44236: BlockingRecv error: recv got EOF from 127.0.0.1:44236 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260927 04:02:45.768522 6004 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:02:45.767287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44242 (local address 127.15.185.125:45391)
0927 04:02:45.767373 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:45.767376 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:45.767510 (+ 134us) server_negotiation.cc:407] Connection header received
0927 04:02:45.767550 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:45.767551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:45.767565 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:45.767794 (+ 229us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:45.768422 (+ 628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44242: BlockingRecv error: recv got EOF from 127.0.0.1:44242 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260927 04:02:46.026538 6004 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:02:46.025811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44248 (local address 127.15.185.125:45391)
0927 04:02:46.025888 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:46.025891 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:46.025916 (+ 25us) server_negotiation.cc:407] Connection header received
0927 04:02:46.026020 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:46.026022 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:46.026036 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:46.026119 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:46.026490 (+ 371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44248: BlockingRecv error: recv got EOF from 127.0.0.1:44248 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 04:02:46.379686 6092 catalog_manager.cc:1259] Loaded cluster ID: 84faabecb13c413d8512aba2d1713625
I20260927 04:02:46.379801 6092 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3d805ac0e6fa4c7986d14d03cf2d4033: loading cluster ID for follower catalog manager: success
I20260927 04:02:46.380633 6092 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3d805ac0e6fa4c7986d14d03cf2d4033: acquiring CA information for follower catalog manager: success
I20260927 04:02:46.380954 6092 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3d805ac0e6fa4c7986d14d03cf2d4033: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 04:02:46.446938 6135 catalog_manager.cc:1259] Loaded cluster ID: 84faabecb13c413d8512aba2d1713625
I20260927 04:02:46.447105 6135 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6a62c49c62384f649489cb5dc8d9dda8: loading cluster ID for follower catalog manager: success
I20260927 04:02:46.447947 6135 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6a62c49c62384f649489cb5dc8d9dda8: acquiring CA information for follower catalog manager: success
I20260927 04:02:46.448276 6135 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6a62c49c62384f649489cb5dc8d9dda8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 04:02:46.542618 6205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:46.540042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44258 (local address 127.15.185.125:45391)
0927 04:02:46.540210 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:46.540211 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:46.540222 (+ 11us) server_negotiation.cc:407] Connection header received
0927 04:02:46.540249 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:46.540250 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:46.540305 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:46.540392 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:02:46.540535 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:46.541047 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:46.541841 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:46.542031 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:46.542127 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:46.542284 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:02:46.542433 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:02:46.542496 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:46.542529 (+ 33us) server_negotiation.cc:299] Negotiation successful
0927 04:02:46.542546 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":94,"threads_started":1}
I20260927 04:02:46.547349 6211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:46.544702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35878 (local address 127.15.185.124:42753)
0927 04:02:46.544831 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:46.544833 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:46.544839 (+ 6us) server_negotiation.cc:407] Connection header received
0927 04:02:46.544885 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:46.544886 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:46.544922 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:46.545019 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:02:46.545167 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:46.545655 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:46.546399 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:46.546620 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:46.546712 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:46.546894 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:02:46.547012 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:02:46.547118 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:46.547227 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 04:02:46.547249 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":57,"threads_started":1}
I20260927 04:02:46.551987 6205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:46.549239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44266 (local address 127.15.185.125:45391)
0927 04:02:46.549301 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:46.549303 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:46.549369 (+ 66us) server_negotiation.cc:407] Connection header received
0927 04:02:46.549472 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:46.549474 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:46.549485 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:46.549537 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:02:46.549697 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:46.550409 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:46.551246 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:46.551456 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:46.551542 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:46.551695 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:02:46.551732 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:02:46.551794 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:46.551864 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 04:02:46.551885 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 04:02:46.552222 6211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:46.549178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35880 (local address 127.15.185.124:42753)
0927 04:02:46.549223 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:46.549225 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:46.549249 (+ 24us) server_negotiation.cc:407] Connection header received
0927 04:02:46.549316 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:46.549317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:46.549329 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:46.549384 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:02:46.549505 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:46.550272 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:46.551311 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:46.551619 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:46.551680 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:46.551868 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:02:46.551900 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:02:46.551940 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:46.552100 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 04:02:46.552125 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 04:02:46.552556 6218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:46.549023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43372 (local address 127.15.185.126:45777)
0927 04:02:46.549366 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:46.549369 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:46.549376 (+ 7us) server_negotiation.cc:407] Connection header received
0927 04:02:46.549420 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:46.549421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:46.549470 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:46.549556 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:02:46.549651 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:46.550180 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:46.551314 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:46.551633 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:46.551686 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:46.551900 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:02:46.552018 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:02:46.552222 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:46.552437 (+ 215us) server_negotiation.cc:299] Negotiation successful
0927 04:02:46.552467 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":50,"threads_started":1}
I20260927 04:02:46.554725 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 5856
I20260927 04:02:46.560045 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 5931
I20260927 04:02:46.565162 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 6006
2026-09-27T04:02:46Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2851 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:02:46 dist-test-slave-glgz krb5kdc[6222](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:02:46 dist-test-slave-glgz krb5kdc[6222](info): set up 2 sockets
Sep 27 04:02:46 dist-test-slave-glgz krb5kdc[6222](info): commencing operation
krb5kdc: starting...
W20260927 04:02:47.631881 16101 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/krb5kdc/test-user.keytab.
Sep 27 04:02:47 dist-test-slave-glgz krb5kdc[6222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:02:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:02:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.185.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab.
Entry for principal kudu/127.15.185.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab.
I20260927 04:02:47.758479 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:38265
--webserver_interface=127.15.185.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:35273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.185.126:38265,127.15.185.125:33025,127.15.185.124:41687
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:02:47.841248 6235 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 04:02:47.841526 6235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:02:47.841557 6235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:02:47.841603 6235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:02:47.842808 6235 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:02:47.842881 6235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:02:47.842904 6235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:02:47.842926 6235 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:02:47.842957 6235 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:02:47.844548 6235 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:35273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.185.126:38265,127.15.185.125:33025,127.15.185.124:41687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.126.keytab
--principal=kudu/127.15.185.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.126:38265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.185.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:02:47.844853 6235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:02:47.845091 6235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:02:47.850174 6240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:47.850190 6243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:47.850203 6241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:47.850574 6235 server_base.cc:1034] running on GCE node
I20260927 04:02:47.850804 6235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:02:47.851099 6235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:02:47.852286 6235 hybrid_clock.cc:648] HybridClock initialized: now 1790481767852264 us; error 34 us; skew 500 ppm
Sep 27 04:02:47 dist-test-slave-glgz krb5kdc[6222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481767, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:02:47.860119 6235 init.cc:377] Logged in from keytab as kudu/127.15.185.126@KRBTEST.COM (short username kudu)
I20260927 04:02:47.860703 6235 webserver.cc:460] Webserver started at http://127.15.185.126:35603/ using document root <none> and password file <none>
I20260927 04:02:47.860916 6235 fs_manager.cc:359] Metadata directory not provided
I20260927 04:02:47.860960 6235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:02:47.861054 6235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:02:47.862160 6235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/minicluster-data/master-0/data/instance:
uuid: "ce506ea54dae424e9dc07eea920574d8"
format_stamp: "Formatted at 2026-09-27 04:02:47 on dist-test-slave-glgz"
server_key: "156903e1d03bc84d1bd04c7b279074a1"
server_key_iv: "64c4551e087c5d8981001dba50f4ae5f"
server_key_version: "encryptionkey@0"
I20260927 04:02:47.862479 6235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/minicluster-data/master-0/wal/instance:
uuid: "ce506ea54dae424e9dc07eea920574d8"
format_stamp: "Formatted at 2026-09-27 04:02:47 on dist-test-slave-glgz"
server_key: "156903e1d03bc84d1bd04c7b279074a1"
server_key_iv: "64c4551e087c5d8981001dba50f4ae5f"
server_key_version: "encryptionkey@0"
I20260927 04:02:47.863970 6235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 04:02:47.864823 6250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:47.865049 6235 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:02:47.865167 6235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/minicluster-data/master-0/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/minicluster-data/master-0/wal
uuid: "ce506ea54dae424e9dc07eea920574d8"
format_stamp: "Formatted at 2026-09-27 04:02:47 on dist-test-slave-glgz"
server_key: "156903e1d03bc84d1bd04c7b279074a1"
server_key_iv: "64c4551e087c5d8981001dba50f4ae5f"
server_key_version: "encryptionkey@0"
I20260927 04:02:47.865252 6235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:02:47.877475 6235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:02:47.880151 6235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:02:47.880313 6235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:02:47.884526 6235 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.126:38265
I20260927 04:02:47.884559 6302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.126:38265 every 8 connection(s)
I20260927 04:02:47.884971 6235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/minicluster-data/master-0/data/info.pb
I20260927 04:02:47.885926 6303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:02:47.887674 6303 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.185.126" port: 38265 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 04:02:47 dist-test-slave-glgz krb5kdc[6222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481767, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.126@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 04:02:47.893921 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 6235
I20260927 04:02:47.894028 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/minicluster-data/master-0/wal/instance
I20260927 04:02:47.894258 16101 external_mini_cluster.cc:1428] Setting key 3f4329cbfa11e26731fa66510dba5e8b
I20260927 04:02:47.897485 6306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:47.888439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:38265 (local address 127.0.0.1:38930)
0927 04:02:47.888622 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:47.888667 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:47.889124 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:47.889396 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:47.889398 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:47.889651 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:02:47.889841 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:47.889845 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:47.890833 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:47.890834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:47.891559 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:47.891562 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:47.892016 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:47.892017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:47.892072 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:47.892543 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:02:47.892552 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:02:47.894102 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:02:47.895921 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:47.895928 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:47.895931 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:47.896126 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:47.896397 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:47.896398 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:47.896398 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:47.896486 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:47.896868 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:02:47.896873 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:02:47.896938 (+ 65us) client_negotiation.cc:769] Sending connection context
0927 04:02:47.897049 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 04:02:47.897293 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":97,"thread_start_us":106,"threads_started":1}
I20260927 04:02:47.897490 6305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:47.888457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38930 (local address 127.15.185.126:38265)
0927 04:02:47.888666 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:47.888675 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:47.888752 (+ 77us) server_negotiation.cc:407] Connection header received
0927 04:02:47.889250 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:47.889252 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:47.889269 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:47.889354 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:47.889925 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:47.890745 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:47.891635 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:47.891940 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:47.892028 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:47.894177 (+ 2149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:47.894185 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:47.894185 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:47.894192 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:47.895815 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:47.896225 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:47.896228 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:47.896229 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:47.896319 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:47.896550 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:47.896551 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:47.896552 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:47.896787 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:47.896878 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:47.897079 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 04:02:47.897225 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":94,"threads_started":1}
I20260927 04:02:47.898378 6303 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.185.125" port: 33025 } has no permanent_uuid. Determining permanent_uuid...
I20260927 04:02:47.898818 6306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:47.898656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.125:33025 (local address 127.0.0.1:57388)
0927 04:02:47.898716 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:47.898775 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.125:33025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 04:02:47.898978 6252 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.185.125:33025: connect: Connection refused (error 111)
I20260927 04:02:47.899521 6306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:47.899397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.125:33025 (local address 127.0.0.1:57396)
0927 04:02:47.899453 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:47.899486 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.125:33025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260927 04:02:47.899647 6303 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.185.125:33025: Network error: Client connection negotiation failed: client connection to 127.15.185.125:33025: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.15.185.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.125.keytab.
Entry for principal kudu/127.15.185.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.125.keytab.
I20260927 04:02:47.923224 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.125:33025
--webserver_interface=127.15.185.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.125.keytab
--principal=kudu/127.15.185.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:35273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.185.126:38265,127.15.185.125:33025,127.15.185.124:41687
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 04:02:47.949008 6303 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.185.125" port: 33025 } attempt: 1
I20260927 04:02:47.949580 6306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:47.949411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.125:33025 (local address 127.0.0.1:57410)
0927 04:02:47.949486 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:47.949538 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.125:33025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 04:02:47.949986 6306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:47.949890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.125:33025 (local address 127.0.0.1:57422)
0927 04:02:47.949937 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:47.949967 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.125:33025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 04:02:47.950119 6303 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.185.125:33025: Network error: Client connection negotiation failed: client connection to 127.15.185.125:33025: connect: Connection refused (error 111)
W20260927 04:02:48.007886 6310 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 04:02:48.008113 6310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:02:48.008132 6310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:02:48.008180 6310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:02:48.009444 6310 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:02:48.009492 6310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:02:48.009507 6310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:02:48.009521 6310 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:02:48.009532 6310 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:02:48.011168 6310 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:35273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.15.185.126:38265,127.15.185.125:33025,127.15.185.124:41687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.125.keytab
--principal=kudu/127.15.185.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.125:33025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.15.185.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:02:48.011442 6310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:02:48.011705 6310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:02:48.017071 6315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:48.017064 6316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:48.017162 6310 server_base.cc:1034] running on GCE node
W20260927 04:02:48.017093 6318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:48.017545 6310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:02:48.017902 6310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:02:48.018294 6303 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.185.125" port: 33025 } attempt: 2
I20260927 04:02:48.018730 6306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:48.018590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.125:33025 (local address 127.0.0.1:57432)
0927 04:02:48.018658 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:48.018693 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.125:33025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 04:02:48.019006 6306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:48.018940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.125:33025 (local address 127.0.0.1:57438)
0927 04:02:48.018971 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:48.018991 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.125:33025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260927 04:02:48.019099 6310 hybrid_clock.cc:648] HybridClock initialized: now 1790481768019076 us; error 55 us; skew 500 ppm
W20260927 04:02:48.019137 6303 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.185.125:33025: Network error: Client connection negotiation failed: client connection to 127.15.185.125:33025: connect: Connection refused (error 111)
Sep 27 04:02:48 dist-test-slave-glgz krb5kdc[6222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481768, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:02:48.021745 6310 init.cc:377] Logged in from keytab as kudu/127.15.185.125@KRBTEST.COM (short username kudu)
I20260927 04:02:48.022253 6310 webserver.cc:460] Webserver started at http://127.15.185.125:41985/ using document root <none> and password file <none>
I20260927 04:02:48.022512 6310 fs_manager.cc:359] Metadata directory not provided
I20260927 04:02:48.022586 6310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:02:48.022753 6310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:02:48.023918 6310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/minicluster-data/master-1/data/instance:
uuid: "5675da163bad4da98f322b523d2ab887"
format_stamp: "Formatted at 2026-09-27 04:02:48 on dist-test-slave-glgz"
server_key: "67154206d5e5efe0873ef8491264b8ea"
server_key_iv: "8c3f940ebdebbaad2864069d73d3eca6"
server_key_version: "encryptionkey@0"
I20260927 04:02:48.024294 6310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/minicluster-data/master-1/wal/instance:
uuid: "5675da163bad4da98f322b523d2ab887"
format_stamp: "Formatted at 2026-09-27 04:02:48 on dist-test-slave-glgz"
server_key: "67154206d5e5efe0873ef8491264b8ea"
server_key_iv: "8c3f940ebdebbaad2864069d73d3eca6"
server_key_version: "encryptionkey@0"
I20260927 04:02:48.025703 6310 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 04:02:48.026420 6325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:48.026592 6310 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 04:02:48.026706 6310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/minicluster-data/master-1/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/minicluster-data/master-1/wal
uuid: "5675da163bad4da98f322b523d2ab887"
format_stamp: "Formatted at 2026-09-27 04:02:48 on dist-test-slave-glgz"
server_key: "67154206d5e5efe0873ef8491264b8ea"
server_key_iv: "8c3f940ebdebbaad2864069d73d3eca6"
server_key_version: "encryptionkey@0"
I20260927 04:02:48.026783 6310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:02:48.037205 6310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:02:48.039664 6310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:02:48.039850 6310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:02:48.043989 6310 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.125:33025
I20260927 04:02:48.044021 6377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.125:33025 every 8 connection(s)
I20260927 04:02:48.044427 6310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/minicluster-data/master-1/data/info.pb
I20260927 04:02:48.045362 6378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:02:48.047150 6378 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.185.126" port: 38265 } has no permanent_uuid. Determining permanent_uuid...
I20260927 04:02:48.048354 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 6310
I20260927 04:02:48.048468 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/minicluster-data/master-1/wal/instance
I20260927 04:02:48.048671 16101 external_mini_cluster.cc:1428] Setting key 4d3f682cffcfc5caad14d263384e92c0
Sep 27 04:02:48 dist-test-slave-glgz krb5kdc[6222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481768, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.125@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.185.124@KRBTEST.COM; defaulting to no policy
I20260927 04:02:48.057086 6305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:48.047891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38938 (local address 127.15.185.126:38265)
0927 04:02:48.047962 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:48.047964 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:48.048272 (+ 308us) server_negotiation.cc:407] Connection header received
0927 04:02:48.048854 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:48.048855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:48.048870 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:48.048920 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:48.049641 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:48.050164 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:48.051228 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:48.051440 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:48.051519 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:48.054099 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:48.054101 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:48.054101 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:48.054130 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:48.055587 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:48.056080 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:48.056082 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:48.056082 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:48.056151 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:48.056414 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:48.056415 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:48.056415 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:48.056535 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:48.056614 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:48.056901 (+ 287us) server_negotiation.cc:299] Negotiation successful
0927 04:02:48.056989 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 04:02:48.057296 6380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:48.047884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:38265 (local address 127.0.0.1:38938)
0927 04:02:48.048102 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:48.048143 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:48.048721 (+ 578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:48.048998 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:48.049001 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:48.049258 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:02:48.049463 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:48.049468 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:48.050263 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:48.050265 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:48.051131 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:48.051133 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:48.051531 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:48.051533 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:48.051660 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:48.052327 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:02:48.052343 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:02:48.053995 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:02:48.055699 (+ 1704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:48.055706 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:48.055708 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:48.055959 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:48.056221 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:48.056222 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:48.056222 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:48.056338 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:48.056604 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:02:48.056620 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:02:48.056766 (+ 146us) client_negotiation.cc:769] Sending connection context
0927 04:02:48.056880 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 04:02:48.057043 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":103,"threads_started":1}
I20260927 04:02:48.057894 6378 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.185.125" port: 33025 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.185.124@KRBTEST.COM" created.
Sep 27 04:02:48 dist-test-slave-glgz krb5kdc[6222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481768, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.125@KRBTEST.COM for kudu/127.15.185.125@KRBTEST.COM
I20260927 04:02:48.065512 6380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:48.058269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.125:33025 (local address 127.0.0.1:57448)
0927 04:02:48.058321 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:48.058332 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:48.058369 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:48.058671 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:48.058672 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:48.058824 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:02:48.058895 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:48.058898 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:48.059870 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:48.059871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:48.060375 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:48.060376 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:48.060903 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:48.060904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:48.060996 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:48.061445 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:02:48.061449 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:02:48.062604 (+ 1155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:02:48.064446 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:48.064447 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:48.064448 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:48.064608 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:48.064831 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:48.064832 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:48.064833 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:48.064886 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:48.065178 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:02:48.065179 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:02:48.065217 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 04:02:48.065282 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 04:02:48.065385 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 04:02:48.065552 6382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:48.058364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57448 (local address 127.15.185.125:33025)
0927 04:02:48.058517 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:48.058522 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:48.058537 (+ 15us) server_negotiation.cc:407] Connection header received
0927 04:02:48.058552 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:48.058554 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:48.058564 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:48.058624 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:48.058983 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:48.059774 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:48.060475 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:48.060811 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:48.060903 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:48.062699 (+ 1796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:48.062707 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:48.062707 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:48.062714 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:48.064345 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:48.064701 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:48.064705 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:48.064705 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:48.064763 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:48.064958 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:48.064959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:48.064959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:48.065108 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:48.065200 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:48.065310 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 04:02:48.065445 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":131,"thread_start_us":86,"threads_started":1}
I20260927 04:02:48.066160 6378 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.185.124" port: 41687 } has no permanent_uuid. Determining permanent_uuid...
I20260927 04:02:48.066538 6380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:48.066393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.124:41687 (local address 127.0.0.1:44456)
0927 04:02:48.066456 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:48.066503 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.124:41687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 04:02:48.066675 6326 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.185.124:41687: connect: Connection refused (error 111)
I20260927 04:02:48.067112 6380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:48.067003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.124:41687 (local address 127.0.0.1:44464)
0927 04:02:48.067066 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:48.067090 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.124:41687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260927 04:02:48.067217 6378 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.185.124:41687: Network error: Client connection negotiation failed: client connection to 127.15.185.124:41687: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.185.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.124.keytab.
Entry for principal kudu/127.15.185.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.124.keytab.
I20260927 04:02:48.075163 16101 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTmmW3q/build/release/bin/kudu
/tmp/dist-test-taskTmmW3q/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.124:41687
--webserver_interface=127.15.185.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.124.keytab
--principal=kudu/127.15.185.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.185.84:35273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.185.126:38265,127.15.185.125:33025,127.15.185.124:41687
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 04:02:48.110280 6303 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.185.125" port: 33025 } attempt: 3
Sep 27 04:02:48 dist-test-slave-glgz krb5kdc[6222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481767, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.126@KRBTEST.COM for kudu/127.15.185.125@KRBTEST.COM
I20260927 04:02:48.116434 6378 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.185.124" port: 41687 } attempt: 1
I20260927 04:02:48.116869 6380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:48.116706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.124:41687 (local address 127.0.0.1:44478)
0927 04:02:48.116801 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:48.116841 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.124:41687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260927 04:02:48.117285 6380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:48.117166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.124:41687 (local address 127.0.0.1:44486)
0927 04:02:48.117224 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:48.117254 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.124:41687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 04:02:48.117383 6378 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.185.124:41687: Network error: Client connection negotiation failed: client connection to 127.15.185.124:41687: connect: Connection refused (error 111)
I20260927 04:02:48.118261 6382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:48.110773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57462 (local address 127.15.185.125:33025)
0927 04:02:48.110845 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:48.110848 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:48.110860 (+ 12us) server_negotiation.cc:407] Connection header received
0927 04:02:48.110909 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:48.110910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:48.110924 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:48.110983 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:48.111407 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:48.111949 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:48.112734 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:48.112980 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:48.113063 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:48.115320 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:48.115322 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:48.115322 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:48.115331 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:48.116955 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:48.117408 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:48.117409 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:48.117410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:48.117470 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:48.117712 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:48.117713 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:48.117713 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:48.117856 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:48.117928 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:48.118098 (+ 170us) server_negotiation.cc:299] Negotiation successful
0927 04:02:48.118203 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 04:02:48.118280 6306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:48.110685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.125:33025 (local address 127.0.0.1:57462)
0927 04:02:48.110753 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:48.110765 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:48.110842 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:48.111011 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:48.111012 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:48.111243 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:02:48.111314 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:48.111316 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:48.112045 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:48.112047 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:48.112623 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:48.112626 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:48.113062 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:48.113062 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:48.113150 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:48.113802 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:02:48.113807 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:02:48.115218 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:02:48.117059 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:48.117060 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:48.117061 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:48.117316 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:48.117532 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:48.117533 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:48.117534 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:48.117610 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:48.117934 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:02:48.117935 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:02:48.117991 (+ 56us) client_negotiation.cc:769] Sending connection context
0927 04:02:48.118063 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 04:02:48.118184 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 04:02:48.118747 6303 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.185.124" port: 41687 } has no permanent_uuid. Determining permanent_uuid...
I20260927 04:02:48.119146 6306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:48.119005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.124:41687 (local address 127.0.0.1:44502)
0927 04:02:48.119081 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:48.119122 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.124:41687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 04:02:48.119500 6306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:48.119418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.124:41687 (local address 127.0.0.1:44504)
0927 04:02:48.119457 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:48.119482 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.124:41687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260927 04:02:48.119589 6303 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.185.124:41687: Network error: Client connection negotiation failed: client connection to 127.15.185.124:41687: connect: Connection refused (error 111)
I20260927 04:02:48.150754 6303 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.185.124" port: 41687 } attempt: 1
I20260927 04:02:48.151358 6306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:48.151188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.124:41687 (local address 127.0.0.1:44506)
0927 04:02:48.151264 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:48.151314 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.124:41687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 04:02:48.151810 6306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:48.151703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.124:41687 (local address 127.0.0.1:44512)
0927 04:02:48.151751 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:48.151783 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.124:41687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 04:02:48.151921 6303 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.185.124:41687: Network error: Client connection negotiation failed: client connection to 127.15.185.124:41687: connect: Connection refused (error 111)
W20260927 04:02:48.161935 6385 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 04:02:48.162158 6385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:02:48.162175 6385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:02:48.162230 6385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:02:48.163492 6385 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:02:48.163539 6385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:02:48.163554 6385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:02:48.163568 6385 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:02:48.163579 6385 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:02:48.165160 6385 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.185.84:35273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.15.185.126:38265,127.15.185.125:33025,127.15.185.124:41687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/krb5kdc/kudu_127.15.185.124.keytab
--principal=kudu/127.15.185.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.185.124:41687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.15.185.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6385
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:02:48.165423 6385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:02:48.165676 6385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:02:48.170853 6390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:48.170900 6393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:48.170853 6391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:48.171303 6385 server_base.cc:1034] running on GCE node
I20260927 04:02:48.171561 6385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:02:48.171868 6385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:02:48.173041 6385 hybrid_clock.cc:648] HybridClock initialized: now 1790481768173025 us; error 48 us; skew 500 ppm
Sep 27 04:02:48 dist-test-slave-glgz krb5kdc[6222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481768, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:02:48.175652 6385 init.cc:377] Logged in from keytab as kudu/127.15.185.124@KRBTEST.COM (short username kudu)
I20260927 04:02:48.176112 6385 webserver.cc:460] Webserver started at http://127.15.185.124:37185/ using document root <none> and password file <none>
I20260927 04:02:48.176359 6385 fs_manager.cc:359] Metadata directory not provided
I20260927 04:02:48.176427 6385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:02:48.176575 6385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:02:48.177695 6385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/minicluster-data/master-2/data/instance:
uuid: "8bd3c979cbb14085b2d858f4f0785f26"
format_stamp: "Formatted at 2026-09-27 04:02:48 on dist-test-slave-glgz"
server_key: "0edbd0e10e9dbf9440acd31360713651"
server_key_iv: "e06c20d1020329ce6af7c42c47b55968"
server_key_version: "encryptionkey@0"
I20260927 04:02:48.178076 6385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/minicluster-data/master-2/wal/instance:
uuid: "8bd3c979cbb14085b2d858f4f0785f26"
format_stamp: "Formatted at 2026-09-27 04:02:48 on dist-test-slave-glgz"
server_key: "0edbd0e10e9dbf9440acd31360713651"
server_key_iv: "e06c20d1020329ce6af7c42c47b55968"
server_key_version: "encryptionkey@0"
I20260927 04:02:48.179530 6385 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 04:02:48.180359 6400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:48.180565 6385 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:02:48.180701 6385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/minicluster-data/master-2/data,/tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/minicluster-data/master-2/wal
uuid: "8bd3c979cbb14085b2d858f4f0785f26"
format_stamp: "Formatted at 2026-09-27 04:02:48 on dist-test-slave-glgz"
server_key: "0edbd0e10e9dbf9440acd31360713651"
server_key_iv: "e06c20d1020329ce6af7c42c47b55968"
server_key_version: "encryptionkey@0"
I20260927 04:02:48.180783 6385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:02:48.185534 6378 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.185.124" port: 41687 } attempt: 2
I20260927 04:02:48.185956 6380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:48.185829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.124:41687 (local address 127.0.0.1:44522)
0927 04:02:48.185888 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:48.185927 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.124:41687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 04:02:48.186348 6380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:48.186250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.124:41687 (local address 127.0.0.1:44526)
0927 04:02:48.186297 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:48.186322 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.185.124:41687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 04:02:48.186432 6378 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.185.124:41687: Network error: Client connection negotiation failed: client connection to 127.15.185.124:41687: connect: Connection refused (error 111)
I20260927 04:02:48.199332 6385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:02:48.201705 6385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:02:48.201882 6385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:02:48.206017 6385 rpc_server.cc:307] RPC server started. Bound to: 127.15.185.124:41687
I20260927 04:02:48.206053 6452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.185.124:41687 every 8 connection(s)
I20260927 04:02:48.206445 6385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/minicluster-data/master-2/data/info.pb
I20260927 04:02:48.207451 6453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:02:48.208987 6453 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.185.126" port: 38265 } has no permanent_uuid. Determining permanent_uuid...
I20260927 04:02:48.210749 16101 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTmmW3q/build/release/bin/kudu as pid 6385
I20260927 04:02:48.210873 16101 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTmmW3q/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481498018407-16101-0/minicluster-data/master-2/wal/instance
I20260927 04:02:48.211153 16101 external_mini_cluster.cc:1428] Setting key 24f1facb24b795be6a86f9394a5b1c7b
Sep 27 04:02:48 dist-test-slave-glgz krb5kdc[6222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481768, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.124@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
Sep 27 04:02:48 dist-test-slave-glgz krb5kdc[6222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.185.124@KRBTEST.COM
I20260927 04:02:48.218936 6305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:48.209688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38952 (local address 127.15.185.126:38265)
0927 04:02:48.209751 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:48.209754 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:48.210089 (+ 335us) server_negotiation.cc:407] Connection header received
0927 04:02:48.210565 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:48.210566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:48.210580 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:48.210659 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:48.211602 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:48.212183 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:48.213336 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:48.213732 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:48.213813 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:48.216008 (+ 2195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:48.216009 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:48.216009 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:48.216018 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:48.217548 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:48.217962 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:48.217963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:48.217964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:48.218024 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:48.218292 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:48.218293 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:48.218293 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:48.218409 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:48.218487 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:48.218763 (+ 276us) server_negotiation.cc:299] Negotiation successful
0927 04:02:48.218844 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 04:02:48.219159 6455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:48.209670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.126:38265 (local address 127.0.0.1:38952)
0927 04:02:48.209947 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:48.209988 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:48.210452 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:48.210715 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:48.210719 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:48.211162 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:02:48.211461 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:48.211469 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:48.212291 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:48.212292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:48.213246 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:48.213249 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:48.213822 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:48.213824 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:48.213929 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:48.214513 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:02:48.214521 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:02:48.215914 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:02:48.217630 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:48.217636 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:48.217639 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:48.217872 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:48.218091 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:48.218092 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:48.218092 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:48.218211 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:48.218478 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:02:48.218484 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:02:48.218627 (+ 143us) client_negotiation.cc:769] Sending connection context
0927 04:02:48.218741 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 04:02:48.218893 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":25,"thread_start_us":145,"threads_started":1}
I20260927 04:02:48.219751 6453 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.185.125" port: 33025 } has no permanent_uuid. Determining permanent_uuid...
I20260927 04:02:48.221143 6457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:48.211637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44534 (local address 127.15.185.124:41687)
0927 04:02:48.211791 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:48.211796 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:48.212398 (+ 602us) server_negotiation.cc:407] Connection header received
0927 04:02:48.212448 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:48.212451 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:48.212480 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:48.212564 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:48.213178 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:48.214149 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:48.214872 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:48.215229 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:48.215303 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:48.217972 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:48.217979 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:48.217979 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:48.217986 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:48.219803 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:48.220286 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:48.220288 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:48.220289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:48.220346 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:48.220558 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:48.220559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:48.220559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:48.220699 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:48.220848 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:48.220930 (+ 82us) server_negotiation.cc:299] Negotiation successful
0927 04:02:48.221003 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":122,"thread_start_us":72,"threads_started":1}
Sep 27 04:02:48 dist-test-slave-glgz krb5kdc[6222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481768, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.124@KRBTEST.COM for kudu/127.15.185.125@KRBTEST.COM
Sep 27 04:02:48 dist-test-slave-glgz krb5kdc[6222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.185.126@KRBTEST.COM
I20260927 04:02:48.227077 6303 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.185.124" port: 41687 } attempt: 2
I20260927 04:02:48.227156 6382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:48.220099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57472 (local address 127.15.185.125:33025)
0927 04:02:48.220152 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:48.220154 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:48.220172 (+ 18us) server_negotiation.cc:407] Connection header received
0927 04:02:48.220198 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:48.220199 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:48.220209 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:48.220280 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:48.220542 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:48.221011 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:48.221976 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:48.222346 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:48.222425 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:48.224417 (+ 1992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:48.224418 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:48.224418 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:48.224424 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:48.226005 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:48.226385 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:48.226386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:48.226386 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:48.226435 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:48.226645 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:48.226645 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:48.226645 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:48.226742 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:48.226826 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:48.226960 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 04:02:48.227070 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 04:02:48.227298 6455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:48.220023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.125:33025 (local address 127.0.0.1:57472)
0927 04:02:48.220069 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:48.220079 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:48.220115 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:48.220283 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:48.220284 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:48.220416 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:02:48.220459 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:48.220460 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:48.221104 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:48.221105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:48.221872 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:48.221875 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:48.222425 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:48.222425 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:48.222477 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:48.222828 (+ 351us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:02:48.222833 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:02:48.224338 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:02:48.226093 (+ 1755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:48.226094 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:48.226095 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:48.226313 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:48.226497 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:48.226497 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:48.226498 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:48.226556 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:48.226819 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:02:48.226820 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:02:48.226863 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 04:02:48.226942 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 04:02:48.227036 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 04:02:48.227954 6453 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.185.124" port: 41687 } has no permanent_uuid. Determining permanent_uuid...
I20260927 04:02:48.230350 6305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:48.221878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38960 (local address 127.15.185.126:38265)
0927 04:02:48.221915 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:48.221916 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:48.221936 (+ 20us) server_negotiation.cc:407] Connection header received
0927 04:02:48.221960 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:48.221960 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:48.221969 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:48.222020 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:48.222598 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:48.223354 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:48.224299 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:48.224546 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:48.224594 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:48.227279 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:48.227280 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:48.227281 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:48.227288 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:48.229061 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:48.229485 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:48.229487 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:48.229487 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:48.229595 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:48.229805 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:48.229806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:48.229807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:48.229915 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:48.230112 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:48.230148 (+ 36us) server_negotiation.cc:299] Negotiation successful
0927 04:02:48.230236 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Sep 27 04:02:48 dist-test-slave-glgz krb5kdc[6222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481767, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.126@KRBTEST.COM for kudu/127.15.185.124@KRBTEST.COM
Sep 27 04:02:48 dist-test-slave-glgz krb5kdc[6222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481768, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.124@KRBTEST.COM for kudu/127.15.185.124@KRBTEST.COM
Sep 27 04:02:48 dist-test-slave-glgz krb5kdc[6222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.185.125@KRBTEST.COM
I20260927 04:02:48.236764 6306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:48.227337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.124:41687 (local address 127.0.0.1:44540)
0927 04:02:48.227405 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:48.227416 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:48.227452 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:48.227778 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:48.227779 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:48.227986 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:02:48.228026 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:48.228028 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:48.228931 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:48.228932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:48.230232 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:48.230235 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:48.230867 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:48.230868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:48.230937 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:48.231356 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:02:48.231361 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:02:48.232613 (+ 1252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:02:48.235300 (+ 2687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:48.235301 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:48.235301 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:48.235804 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:48.236032 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:48.236033 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:48.236033 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:48.236102 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:48.236433 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:02:48.236435 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:02:48.236480 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 04:02:48.236557 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 04:02:48.236657 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 04:02:48.236765 6457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:48.227371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44540 (local address 127.15.185.124:41687)
0927 04:02:48.227650 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:48.227652 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:48.227662 (+ 10us) server_negotiation.cc:407] Connection header received
0927 04:02:48.227682 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:48.227683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:48.227692 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:48.227750 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:48.228120 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:48.228841 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:48.230322 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:48.230640 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:48.230725 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:48.232694 (+ 1969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:48.232696 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:48.232697 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:48.232702 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:48.235214 (+ 2512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:48.235893 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:48.235894 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:48.235895 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:48.235954 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:48.236183 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:48.236184 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:48.236184 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:48.236373 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:48.236444 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:48.236599 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 04:02:48.236681 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259}
I20260927 04:02:48.238150 6455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:48.228234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.124:41687 (local address 127.0.0.1:44548)
0927 04:02:48.229162 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:48.229174 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:48.229220 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:48.229384 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:48.229385 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:48.229528 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:02:48.229567 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:48.229569 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:48.230530 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:48.230530 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:48.231308 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:48.231311 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:48.231656 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:48.231657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:48.231703 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:48.232611 (+ 908us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:02:48.232616 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:02:48.233963 (+ 1347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:02:48.236365 (+ 2402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:48.236366 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:48.236367 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:48.236571 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:48.237213 (+ 642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:48.237214 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:48.237214 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:48.237280 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:48.237782 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:02:48.237783 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:02:48.237826 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 04:02:48.237906 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 04:02:48.238008 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":912}
I20260927 04:02:48.238467 6458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:48.228277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44548 (local address 127.15.185.124:41687)
0927 04:02:48.228437 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:48.228439 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:48.229230 (+ 791us) server_negotiation.cc:407] Connection header received
0927 04:02:48.229263 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:48.229264 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:48.229275 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:48.229371 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:48.229698 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:48.230461 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:48.231375 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:48.231567 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:48.231655 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:48.234084 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:48.234085 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:48.234085 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:48.234091 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:48.236287 (+ 2196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:48.237071 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:48.237072 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:48.237073 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:48.237134 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:48.237566 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:48.237567 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:48.237567 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:48.237692 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:48.237769 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:48.237949 (+ 180us) server_negotiation.cc:299] Negotiation successful
0927 04:02:48.238074 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":141,"thread_start_us":94,"threads_started":1}
I20260927 04:02:48.238644 6303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce506ea54dae424e9dc07eea920574d8: Bootstrap starting.
I20260927 04:02:48.239468 6303 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce506ea54dae424e9dc07eea920574d8: Neither blocks nor log segments found. Creating new log.
I20260927 04:02:48.239480 6382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:48.230901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57474 (local address 127.15.185.125:33025)
0927 04:02:48.230938 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:48.230940 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:48.231158 (+ 218us) server_negotiation.cc:407] Connection header received
0927 04:02:48.231231 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:48.231232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:48.231241 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:48.231298 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:48.231934 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:48.232510 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:48.233362 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:48.233692 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:48.233757 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:48.236400 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:48.236401 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:48.236401 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:48.236409 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:48.238187 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:48.238624 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:48.238626 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:48.238626 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:48.238683 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:48.238895 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:48.238897 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:48.238897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:48.239072 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:48.239253 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:48.239294 (+ 41us) server_negotiation.cc:299] Negotiation successful
0927 04:02:48.239375 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 04:02:48.239760 6303 log.cc:824] T 00000000000000000000000000000000 P ce506ea54dae424e9dc07eea920574d8: Log is configured to *not* fsync() on all Append() calls
I20260927 04:02:48.240456 6453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bd3c979cbb14085b2d858f4f0785f26: Bootstrap starting.
I20260927 04:02:48.240526 6303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce506ea54dae424e9dc07eea920574d8: No bootstrap required, opened a new log
I20260927 04:02:48.241060 6453 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8bd3c979cbb14085b2d858f4f0785f26: Neither blocks nor log segments found. Creating new log.
I20260927 04:02:48.241330 6453 log.cc:824] T 00000000000000000000000000000000 P 8bd3c979cbb14085b2d858f4f0785f26: Log is configured to *not* fsync() on all Append() calls
I20260927 04:02:48.242067 6453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bd3c979cbb14085b2d858f4f0785f26: No bootstrap required, opened a new log
I20260927 04:02:48.242405 6303 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce506ea54dae424e9dc07eea920574d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce506ea54dae424e9dc07eea920574d8" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 38265 } } peers { permanent_uuid: "5675da163bad4da98f322b523d2ab887" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 33025 } } peers { permanent_uuid: "8bd3c979cbb14085b2d858f4f0785f26" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 41687 } }
I20260927 04:02:48.242574 6303 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce506ea54dae424e9dc07eea920574d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce506ea54dae424e9dc07eea920574d8, State: Initialized, Role: FOLLOWER
I20260927 04:02:48.242664 6303 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce506ea54dae424e9dc07eea920574d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce506ea54dae424e9dc07eea920574d8" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 38265 } } peers { permanent_uuid: "5675da163bad4da98f322b523d2ab887" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 33025 } } peers { permanent_uuid: "8bd3c979cbb14085b2d858f4f0785f26" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 41687 } }
I20260927 04:02:48.242939 6461 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce506ea54dae424e9dc07eea920574d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce506ea54dae424e9dc07eea920574d8" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 38265 } } peers { permanent_uuid: "5675da163bad4da98f322b523d2ab887" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 33025 } } peers { permanent_uuid: "8bd3c979cbb14085b2d858f4f0785f26" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 41687 } } }
I20260927 04:02:48.243024 6303 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce506ea54dae424e9dc07eea920574d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:02:48.243155 6461 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce506ea54dae424e9dc07eea920574d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 04:02:48.243378 16101 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 04:02:48.244206 6453 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8bd3c979cbb14085b2d858f4f0785f26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce506ea54dae424e9dc07eea920574d8" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 38265 } } peers { permanent_uuid: "5675da163bad4da98f322b523d2ab887" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 33025 } } peers { permanent_uuid: "8bd3c979cbb14085b2d858f4f0785f26" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 41687 } }
I20260927 04:02:48.244360 6453 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8bd3c979cbb14085b2d858f4f0785f26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bd3c979cbb14085b2d858f4f0785f26, State: Initialized, Role: FOLLOWER
I20260927 04:02:48.244445 6453 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8bd3c979cbb14085b2d858f4f0785f26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce506ea54dae424e9dc07eea920574d8" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 38265 } } peers { permanent_uuid: "5675da163bad4da98f322b523d2ab887" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 33025 } } peers { permanent_uuid: "8bd3c979cbb14085b2d858f4f0785f26" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 41687 } }
I20260927 04:02:48.244864 6453 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8bd3c979cbb14085b2d858f4f0785f26 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:02:48.244819 6473 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bd3c979cbb14085b2d858f4f0785f26 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce506ea54dae424e9dc07eea920574d8" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 38265 } } peers { permanent_uuid: "5675da163bad4da98f322b523d2ab887" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 33025 } } peers { permanent_uuid: "8bd3c979cbb14085b2d858f4f0785f26" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 41687 } } }
I20260927 04:02:48.244925 6473 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bd3c979cbb14085b2d858f4f0785f26 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 04:02:48.245144 6472 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ce506ea54dae424e9dc07eea920574d8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260927 04:02:48.244468 6277 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "ce506ea54dae424e9dc07eea920574d8"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38960
W20260927 04:02:48.245252 6472 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 04:02:48.245299 6277 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce506ea54dae424e9dc07eea920574d8 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 04:02:48.245471 6277 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce506ea54dae424e9dc07eea920574d8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:02:48.246443 6277 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce506ea54dae424e9dc07eea920574d8 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce506ea54dae424e9dc07eea920574d8" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 38265 } } peers { permanent_uuid: "5675da163bad4da98f322b523d2ab887" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 33025 } } peers { permanent_uuid: "8bd3c979cbb14085b2d858f4f0785f26" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 41687 } }
I20260927 04:02:48.246702 6277 leader_election.cc:288] T 00000000000000000000000000000000 P ce506ea54dae424e9dc07eea920574d8 [CANDIDATE]: Term 1 election: Requested vote from peers 5675da163bad4da98f322b523d2ab887 (127.15.185.125:33025), 8bd3c979cbb14085b2d858f4f0785f26 (127.15.185.124:41687)
I20260927 04:02:48.245802 6235 master_runner.cc:186] Error getting master registration for 127.15.185.125:33025: OK, instance_id { permanent_uuid: "5675da163bad4da98f322b523d2ab887" instance_seqno: 1790481768042721 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 04:02:48.248796 6235 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 04:02:48.246695 6485 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8bd3c979cbb14085b2d858f4f0785f26: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 04:02:48.249754 6485 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 04:02:48.247008 6427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ce506ea54dae424e9dc07eea920574d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8bd3c979cbb14085b2d858f4f0785f26"
I20260927 04:02:48.249965 6427 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8bd3c979cbb14085b2d858f4f0785f26 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:02:48.251001 6427 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8bd3c979cbb14085b2d858f4f0785f26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce506ea54dae424e9dc07eea920574d8 in term 1.
I20260927 04:02:48.252310 6251 leader_election.cc:302] T 00000000000000000000000000000000 P ce506ea54dae424e9dc07eea920574d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8bd3c979cbb14085b2d858f4f0785f26, ce506ea54dae424e9dc07eea920574d8; no voters:
I20260927 04:02:48.252451 6461 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce506ea54dae424e9dc07eea920574d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:02:48.252607 6461 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce506ea54dae424e9dc07eea920574d8 [term 1 LEADER]: Becoming Leader. State: Replica: ce506ea54dae424e9dc07eea920574d8, State: Running, Role: LEADER
I20260927 04:02:48.252692 6461 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce506ea54dae424e9dc07eea920574d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce506ea54dae424e9dc07eea920574d8" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 38265 } } peers { permanent_uuid: "5675da163bad4da98f322b523d2ab887" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 33025 } } peers { permanent_uuid: "8bd3c979cbb14085b2d858f4f0785f26" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 41687 } }
I20260927 04:02:48.253072 6461 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce506ea54dae424e9dc07eea920574d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce506ea54dae424e9dc07eea920574d8. Latest consensus state: current_term: 1 leader_uuid: "ce506ea54dae424e9dc07eea920574d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce506ea54dae424e9dc07eea920574d8" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 38265 } } peers { permanent_uuid: "5675da163bad4da98f322b523d2ab887" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 33025 } } peers { permanent_uuid: "8bd3c979cbb14085b2d858f4f0785f26" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 41687 } } }
I20260927 04:02:48.253168 6461 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce506ea54dae424e9dc07eea920574d8 [sys.catalog]: This master's current role is: LEADER
I20260927 04:02:48.252353 6385 master_runner.cc:186] Error getting master registration for 127.15.185.125:33025: OK, instance_id { permanent_uuid: "5675da163bad4da98f322b523d2ab887" instance_seqno: 1790481768042721 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 04:02:48.254830 6385 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 04:02:48.257411 6496 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:02:48.258090 6496 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:02:48.260810 6458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:48.247966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44560 (local address 127.15.185.124:41687)
0927 04:02:48.248001 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:48.248002 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:48.249395 (+ 1393us) server_negotiation.cc:407] Connection header received
0927 04:02:48.249463 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:48.249464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:48.249476 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:48.249537 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:48.250571 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:48.251402 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:48.252630 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:48.252951 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:48.252995 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:48.256810 (+ 3815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:48.256811 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:48.256812 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:48.256826 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:48.258518 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:48.259113 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:48.259114 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:48.259115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:48.259175 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:48.260175 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:48.260176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:48.260177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:48.260321 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:48.260415 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:48.260631 (+ 216us) server_negotiation.cc:299] Negotiation successful
0927 04:02:48.260718 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 04:02:48.261679 6305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:48.247835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38968 (local address 127.15.185.126:38265)
0927 04:02:48.247874 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:48.247876 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:48.249871 (+ 1995us) server_negotiation.cc:407] Connection header received
0927 04:02:48.249970 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:48.249971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:48.249982 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:48.250011 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:48.250716 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:48.251422 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:48.255204 (+ 3782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:48.255499 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:48.255542 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:48.257463 (+ 1921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:48.257464 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:48.257464 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:48.257472 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:48.259949 (+ 2477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:48.260556 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:48.260558 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:48.260558 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:48.260631 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:48.260936 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:48.260937 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:48.260937 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:48.261079 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:48.261151 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:48.261519 (+ 368us) server_negotiation.cc:299] Negotiation successful
0927 04:02:48.261597 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 04:02:48.263226 6427 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8bd3c979cbb14085b2d858f4f0785f26 [term 1 FOLLOWER]: Refusing update from remote peer ce506ea54dae424e9dc07eea920574d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:02:48.263594 6461 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ce506ea54dae424e9dc07eea920574d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8bd3c979cbb14085b2d858f4f0785f26" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 41687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:02:48.265914 6473 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bd3c979cbb14085b2d858f4f0785f26 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce506ea54dae424e9dc07eea920574d8. Latest consensus state: current_term: 1 leader_uuid: "ce506ea54dae424e9dc07eea920574d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce506ea54dae424e9dc07eea920574d8" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 38265 } } peers { permanent_uuid: "5675da163bad4da98f322b523d2ab887" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 33025 } } peers { permanent_uuid: "8bd3c979cbb14085b2d858f4f0785f26" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 41687 } } }
I20260927 04:02:48.266005 6473 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bd3c979cbb14085b2d858f4f0785f26 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 04:02:48.266850 6382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:48.249629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57478 (local address 127.15.185.125:33025)
0927 04:02:48.249846 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:48.249848 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:48.255948 (+ 6100us) server_negotiation.cc:407] Connection header received
0927 04:02:48.256050 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:48.256052 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:48.256063 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:48.256098 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:48.257832 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:48.258559 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:48.261344 (+ 2785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:48.261668 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:48.261711 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:48.263758 (+ 2047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:48.263759 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:48.263760 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:48.263767 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:48.265353 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:48.265895 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:48.265896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:48.265897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:48.265950 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:48.266228 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:48.266229 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:48.266230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:48.266381 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:48.266492 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:48.266677 (+ 185us) server_negotiation.cc:299] Negotiation successful
0927 04:02:48.266759 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193}
I20260927 04:02:48.269486 6498 mvcc.cc:205] Tried to move back new op lower bound from 7333813322803298304 to 7333813322763640832. Current Snapshot: MvccSnapshot[applied={T|T < 7333813322803298304}]
I20260927 04:02:48.269690 6473 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bd3c979cbb14085b2d858f4f0785f26 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ce506ea54dae424e9dc07eea920574d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce506ea54dae424e9dc07eea920574d8" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 38265 } } peers { permanent_uuid: "5675da163bad4da98f322b523d2ab887" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 33025 } } peers { permanent_uuid: "8bd3c979cbb14085b2d858f4f0785f26" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 41687 } } }
I20260927 04:02:48.269780 6473 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bd3c979cbb14085b2d858f4f0785f26 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 04:02:48.270052 6494 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce506ea54dae424e9dc07eea920574d8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ce506ea54dae424e9dc07eea920574d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce506ea54dae424e9dc07eea920574d8" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 38265 } } peers { permanent_uuid: "5675da163bad4da98f322b523d2ab887" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 33025 } } peers { permanent_uuid: "8bd3c979cbb14085b2d858f4f0785f26" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 41687 } } }
I20260927 04:02:48.270151 6494 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce506ea54dae424e9dc07eea920574d8 [sys.catalog]: This master's current role is: LEADER
I20260927 04:02:48.270983 6496 catalog_manager.cc:1347] Generated new cluster ID: a07c7d878b6f42b5a9e25eb8ca02a478
I20260927 04:02:48.271034 6496 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:02:48.277581 6378 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.185.124" port: 41687 } attempt: 3
Sep 27 04:02:48 dist-test-slave-glgz krb5kdc[6222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481768, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.185.125@KRBTEST.COM for kudu/127.15.185.124@KRBTEST.COM
I20260927 04:02:48.285853 6380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:48.277854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.185.124:41687 (local address 127.0.0.1:44566)
0927 04:02:48.277967 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:48.277978 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:48.278027 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:48.278195 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:48.278196 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:48.278386 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:02:48.278439 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:48.278441 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:48.279403 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:48.279405 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:48.280209 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:48.280212 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:48.280713 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:48.280714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:48.280775 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:48.281298 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:02:48.281303 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:02:48.282824 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:02:48.284748 (+ 1924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:48.284749 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:48.284750 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:48.284977 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:48.285189 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:48.285190 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:48.285190 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:48.285246 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:48.285544 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:02:48.285546 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:02:48.285592 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 04:02:48.285650 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 04:02:48.285747 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96}
I20260927 04:02:48.286441 6458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:48.277928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44566 (local address 127.15.185.124:41687)
0927 04:02:48.277970 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:48.277973 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:48.278013 (+ 40us) server_negotiation.cc:407] Connection header received
0927 04:02:48.278103 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:48.278104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:48.278124 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:48.278172 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:48.278533 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:48.279315 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:48.280305 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:48.280615 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:48.280695 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:48.282912 (+ 2217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:48.282913 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:48.282914 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:48.282921 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:48.284660 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:48.285062 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:48.285064 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:48.285064 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:48.285121 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:48.285315 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:48.285316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:48.285316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:48.285463 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:48.285518 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:48.286281 (+ 763us) server_negotiation.cc:299] Negotiation successful
0927 04:02:48.286357 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 04:02:48.288080 6378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5675da163bad4da98f322b523d2ab887: Bootstrap starting.
I20260927 04:02:48.288764 6378 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5675da163bad4da98f322b523d2ab887: Neither blocks nor log segments found. Creating new log.
I20260927 04:02:48.289016 6378 log.cc:824] T 00000000000000000000000000000000 P 5675da163bad4da98f322b523d2ab887: Log is configured to *not* fsync() on all Append() calls
I20260927 04:02:48.296176 6378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5675da163bad4da98f322b523d2ab887: No bootstrap required, opened a new log
I20260927 04:02:48.298081 6378 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5675da163bad4da98f322b523d2ab887 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce506ea54dae424e9dc07eea920574d8" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 38265 } } peers { permanent_uuid: "5675da163bad4da98f322b523d2ab887" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 33025 } } peers { permanent_uuid: "8bd3c979cbb14085b2d858f4f0785f26" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 41687 } }
I20260927 04:02:48.298213 6378 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5675da163bad4da98f322b523d2ab887 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5675da163bad4da98f322b523d2ab887, State: Initialized, Role: FOLLOWER
I20260927 04:02:48.298291 6378 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5675da163bad4da98f322b523d2ab887 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce506ea54dae424e9dc07eea920574d8" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 38265 } } peers { permanent_uuid: "5675da163bad4da98f322b523d2ab887" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 33025 } } peers { permanent_uuid: "8bd3c979cbb14085b2d858f4f0785f26" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 41687 } }
I20260927 04:02:48.298601 6378 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5675da163bad4da98f322b523d2ab887 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:02:48.299244 6352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ce506ea54dae424e9dc07eea920574d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5675da163bad4da98f322b523d2ab887"
I20260927 04:02:48.299351 6352 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5675da163bad4da98f322b523d2ab887 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:02:48.300135 6352 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5675da163bad4da98f322b523d2ab887 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce506ea54dae424e9dc07eea920574d8 in term 1.
I20260927 04:02:48.300421 6351 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5675da163bad4da98f322b523d2ab887 [term 1 FOLLOWER]: Refusing update from remote peer ce506ea54dae424e9dc07eea920574d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:02:48.300704 6502 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ce506ea54dae424e9dc07eea920574d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5675da163bad4da98f322b523d2ab887" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 33025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:02:48.303751 6504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5675da163bad4da98f322b523d2ab887 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce506ea54dae424e9dc07eea920574d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce506ea54dae424e9dc07eea920574d8" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 38265 } } peers { permanent_uuid: "5675da163bad4da98f322b523d2ab887" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 33025 } } peers { permanent_uuid: "8bd3c979cbb14085b2d858f4f0785f26" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 41687 } } }
I20260927 04:02:48.303925 6504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5675da163bad4da98f322b523d2ab887 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 04:02:48.303977 6504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5675da163bad4da98f322b523d2ab887 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce506ea54dae424e9dc07eea920574d8. Latest consensus state: current_term: 1 leader_uuid: "ce506ea54dae424e9dc07eea920574d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce506ea54dae424e9dc07eea920574d8" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 38265 } } peers { permanent_uuid: "5675da163bad4da98f322b523d2ab887" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 33025 } } peers { permanent_uuid: "8bd3c979cbb14085b2d858f4f0785f26" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 41687 } } }
I20260927 04:02:48.304023 6504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5675da163bad4da98f322b523d2ab887 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 04:02:48.304060 6504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5675da163bad4da98f322b523d2ab887 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ce506ea54dae424e9dc07eea920574d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce506ea54dae424e9dc07eea920574d8" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 38265 } } peers { permanent_uuid: "5675da163bad4da98f322b523d2ab887" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 33025 } } peers { permanent_uuid: "8bd3c979cbb14085b2d858f4f0785f26" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 41687 } } }
I20260927 04:02:48.304103 6504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5675da163bad4da98f322b523d2ab887 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 04:02:48.305627 6502 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce506ea54dae424e9dc07eea920574d8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ce506ea54dae424e9dc07eea920574d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce506ea54dae424e9dc07eea920574d8" member_type: VOTER last_known_addr { host: "127.15.185.126" port: 38265 } } peers { permanent_uuid: "5675da163bad4da98f322b523d2ab887" member_type: VOTER last_known_addr { host: "127.15.185.125" port: 33025 } } peers { permanent_uuid: "8bd3c979cbb14085b2d858f4f0785f26" member_type: VOTER last_known_addr { host: "127.15.185.124" port: 41687 } } }
I20260927 04:02:48.305728 6502 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce506ea54dae424e9dc07eea920574d8 [sys.catalog]: This master's current role is: LEADER
I20260927 04:02:48.306582 6496 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:02:48.307387 6496 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:02:48.311466 6496 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce506ea54dae424e9dc07eea920574d8: Generated new TSK 0
I20260927 04:02:48.311769 6496 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 04:02:48.315527 6516 mvcc.cc:205] Tried to move back new op lower bound from 7333813322803298304 to 7333813322763640832. Current Snapshot: MvccSnapshot[applied={T|T < 7333813322803298304}]
I20260927 04:02:48.321900 6519 catalog_manager.cc:1259] Loaded cluster ID: a07c7d878b6f42b5a9e25eb8ca02a478
I20260927 04:02:48.321969 6519 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5675da163bad4da98f322b523d2ab887: loading cluster ID for follower catalog manager: success
I20260927 04:02:48.323180 6519 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5675da163bad4da98f322b523d2ab887: acquiring CA information for follower catalog manager: success
I20260927 04:02:48.323540 6519 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5675da163bad4da98f322b523d2ab887: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 04:02:48.341413 6305 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:02:48.339700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38976 (local address 127.15.185.126:38265)
0927 04:02:48.339748 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:48.339750 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:48.340728 (+ 978us) server_negotiation.cc:407] Connection header received
0927 04:02:48.340841 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:48.340842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:48.340853 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:48.340884 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:48.341328 (+ 444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38976: BlockingRecv error: recv got EOF from 127.0.0.1:38976 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260927 04:02:48.342212 6382 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:02:48.340120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57482 (local address 127.15.185.125:33025)
0927 04:02:48.340163 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:48.340165 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:48.341716 (+ 1551us) server_negotiation.cc:407] Connection header received
0927 04:02:48.341814 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:48.341815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:48.341826 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:48.341855 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:48.342155 (+ 300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57482: BlockingRecv error: recv got EOF from 127.0.0.1:57482 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260927 04:02:48.342933 6458 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:02:48.340457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44570 (local address 127.15.185.124:41687)
0927 04:02:48.340496 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:48.340498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:48.342445 (+ 1947us) server_negotiation.cc:407] Connection header received
0927 04:02:48.342541 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:48.342542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:48.342552 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:48.342592 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:48.342876 (+ 284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44570: BlockingRecv error: recv got EOF from 127.0.0.1:44570 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 04:02:48.348340 6382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:48.344550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57492 (local address 127.15.185.125:33025)
0927 04:02:48.344691 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:48.344693 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:48.344940 (+ 247us) server_negotiation.cc:407] Connection header received
0927 04:02:48.345079 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:48.345079 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:48.345120 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:48.345167 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:02:48.345385 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:48.346164 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:48.347348 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:48.347677 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:48.347718 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:48.347963 (+ 245us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:02:48.348089 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:02:48.348138 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:48.348250 (+ 112us) server_negotiation.cc:299] Negotiation successful
0927 04:02:48.348268 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
W20260927 04:02:48.351516 6458 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:02:48.350769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44578 (local address 127.15.185.124:41687)
0927 04:02:48.350805 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:48.350807 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:48.350958 (+ 151us) server_negotiation.cc:407] Connection header received
0927 04:02:48.351090 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:48.351091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:48.351111 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:48.351143 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:48.351466 (+ 323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44578: BlockingRecv error: recv got EOF from 127.0.0.1:44578 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 04:02:48.356984 6382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:48.353723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57496 (local address 127.15.185.125:33025)
0927 04:02:48.353756 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:48.353758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:48.353965 (+ 207us) server_negotiation.cc:407] Connection header received
0927 04:02:48.354055 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:48.354056 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:48.354066 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:48.354093 (+ 27us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:02:48.354258 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:48.355012 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:48.356179 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:48.356481 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:48.356522 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:48.356731 (+ 209us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:02:48.356765 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:02:48.356792 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:48.356894 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 04:02:48.356913 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
W20260927 04:02:48.360109 6458 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:02:48.359388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44590 (local address 127.15.185.124:41687)
0927 04:02:48.359426 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:48.359428 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:48.359604 (+ 176us) server_negotiation.cc:407] Connection header received
0927 04:02:48.359696 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:48.359697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:48.359707 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:48.359736 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:48.360056 (+ 320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44590: BlockingRecv error: recv got EOF from 127.0.0.1:44590 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 04:02:48.366573 6382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:48.363246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57502 (local address 127.15.185.125:33025)
0927 04:02:48.363281 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:48.363283 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:48.363505 (+ 222us) server_negotiation.cc:407] Connection header received
0927 04:02:48.363615 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:48.363616 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:48.363625 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:48.363652 (+ 27us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:02:48.363826 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:48.364575 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:48.365779 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:48.366074 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:48.366114 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:48.366309 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:02:48.366342 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:02:48.366369 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:48.366468 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 04:02:48.366487 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260927 04:02:48.369740 6458 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:02:48.368956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44604 (local address 127.15.185.124:41687)
0927 04:02:48.368992 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:48.368994 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:48.369213 (+ 219us) server_negotiation.cc:407] Connection header received
0927 04:02:48.369306 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:48.369307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:48.369318 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:48.369347 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:48.369687 (+ 340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44604: BlockingRecv error: recv got EOF from 127.0.0.1:44604 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 04:02:48.378376 6382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:48.374925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57508 (local address 127.15.185.125:33025)
0927 04:02:48.374974 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:48.374976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:48.375290 (+ 314us) server_negotiation.cc:407] Connection header received
0927 04:02:48.375405 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:48.375406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:48.375417 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:48.375446 (+ 29us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:02:48.375632 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:48.376373 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:48.377594 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:48.377897 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:48.377935 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:48.378122 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:02:48.378154 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:02:48.378181 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:48.378280 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 04:02:48.378296 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260927 04:02:48.381376 6458 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:02:48.380647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44618 (local address 127.15.185.124:41687)
0927 04:02:48.380687 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:48.380689 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:48.380851 (+ 162us) server_negotiation.cc:407] Connection header received
0927 04:02:48.380945 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:48.380946 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:48.380956 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:48.380986 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:48.381323 (+ 337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44618: BlockingRecv error: recv got EOF from 127.0.0.1:44618 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 04:02:48.394148 6382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:48.390665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57522 (local address 127.15.185.125:33025)
0927 04:02:48.390706 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:48.390708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:48.390951 (+ 243us) server_negotiation.cc:407] Connection header received
0927 04:02:48.391126 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:48.391128 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:48.391139 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:48.391171 (+ 32us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:02:48.391376 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:48.392152 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:48.393314 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:48.393638 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:48.393678 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:48.393874 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:02:48.393908 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:02:48.393934 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:48.394035 (+ 101us) server_negotiation.cc:299] Negotiation successful
0927 04:02:48.394053 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260927 04:02:48.397369 6458 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:02:48.396592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44630 (local address 127.15.185.124:41687)
0927 04:02:48.396632 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:48.396635 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:48.396812 (+ 177us) server_negotiation.cc:407] Connection header received
0927 04:02:48.396910 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:48.396911 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:48.396922 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:48.396952 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:48.397311 (+ 359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44630: BlockingRecv error: recv got EOF from 127.0.0.1:44630 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 04:02:48.418260 6382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:48.414726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57538 (local address 127.15.185.125:33025)
0927 04:02:48.414774 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:48.414776 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:48.415110 (+ 334us) server_negotiation.cc:407] Connection header received
0927 04:02:48.415227 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:48.415228 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:48.415239 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:48.415268 (+ 29us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:02:48.415496 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:48.416259 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:48.417427 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:48.417709 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:48.417755 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:48.417966 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:02:48.418000 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:02:48.418029 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:48.418155 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 04:02:48.418176 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260927 04:02:48.421547 6458 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:02:48.420726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44636 (local address 127.15.185.124:41687)
0927 04:02:48.420769 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:48.420771 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:48.420955 (+ 184us) server_negotiation.cc:407] Connection header received
0927 04:02:48.421069 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:48.421070 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:48.421092 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:48.421123 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:48.421491 (+ 368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44636: BlockingRecv error: recv got EOF from 127.0.0.1:44636 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 04:02:48.458372 6382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:48.455008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57544 (local address 127.15.185.125:33025)
0927 04:02:48.455080 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:48.455090 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:48.455267 (+ 177us) server_negotiation.cc:407] Connection header received
0927 04:02:48.455309 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:48.455310 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:48.455323 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:48.455395 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:02:48.455562 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:48.456390 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:48.457500 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:48.457830 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:48.457909 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:48.458091 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:02:48.458130 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:02:48.458186 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:48.458260 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 04:02:48.458280 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260927 04:02:48.463913 6458 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:02:48.460476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44640 (local address 127.15.185.124:41687)
0927 04:02:48.460532 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:48.460534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:48.463269 (+ 2735us) server_negotiation.cc:407] Connection header received
0927 04:02:48.463315 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:48.463316 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:48.463336 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:48.463411 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:48.463847 (+ 436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44640: BlockingRecv error: recv got EOF from 127.0.0.1:44640 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 04:02:48.531447 6382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:48.529139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57546 (local address 127.15.185.125:33025)
0927 04:02:48.529244 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:48.529247 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:48.529266 (+ 19us) server_negotiation.cc:407] Connection header received
0927 04:02:48.529335 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:48.529336 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:48.529346 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:48.529416 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:02:48.529609 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:48.530085 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:48.530847 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:48.531031 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:48.531110 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:48.531250 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:02:48.531278 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:02:48.531317 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:48.531372 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 04:02:48.531387 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260927 04:02:48.533898 6458 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:02:48.533249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44654 (local address 127.15.185.124:41687)
0927 04:02:48.533329 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:48.533331 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:48.533393 (+ 62us) server_negotiation.cc:407] Connection header received
0927 04:02:48.533451 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:48.533452 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:48.533461 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:48.533527 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:48.533844 (+ 317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44654: BlockingRecv error: recv got EOF from 127.0.0.1:44654 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 04:02:48.665733 6382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:48.663187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57562 (local address 127.15.185.125:33025)
0927 04:02:48.663256 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:48.663259 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:48.663338 (+ 79us) server_negotiation.cc:407] Connection header received
0927 04:02:48.663441 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:48.663442 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:48.663454 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:48.663527 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:02:48.663683 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:48.664178 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:48.664998 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:48.665196 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:48.665273 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:48.665432 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:02:48.665467 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:02:48.665538 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:48.665607 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 04:02:48.665627 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260927 04:02:48.668453 6458 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:02:48.667725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44668 (local address 127.15.185.124:41687)
0927 04:02:48.667795 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:48.667798 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:48.667826 (+ 28us) server_negotiation.cc:407] Connection header received
0927 04:02:48.667915 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:48.667917 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:48.667928 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:48.668000 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:48.668391 (+ 391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44668: BlockingRecv error: recv got EOF from 127.0.0.1:44668 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 04:02:48.928246 6382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:48.925776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57568 (local address 127.15.185.125:33025)
0927 04:02:48.925852 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:48.925854 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:48.925899 (+ 45us) server_negotiation.cc:407] Connection header received
0927 04:02:48.925992 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:48.925993 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:48.926004 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:48.926075 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:02:48.926246 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:48.926733 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:48.927536 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:48.927740 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:48.927819 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:48.927970 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:02:48.928010 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:02:48.928066 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:48.928130 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 04:02:48.928150 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260927 04:02:48.930475 6458 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:02:48.929872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44674 (local address 127.15.185.124:41687)
0927 04:02:48.929945 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:48.929948 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:48.929959 (+ 11us) server_negotiation.cc:407] Connection header received
0927 04:02:48.930028 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:48.930029 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:48.930041 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:48.930085 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:48.930421 (+ 336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44674: BlockingRecv error: recv got EOF from 127.0.0.1:44674 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 04:02:49.250312 6485 catalog_manager.cc:1259] Loaded cluster ID: a07c7d878b6f42b5a9e25eb8ca02a478
I20260927 04:02:49.250375 6485 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8bd3c979cbb14085b2d858f4f0785f26: loading cluster ID for follower catalog manager: success
I20260927 04:02:49.251282 6485 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8bd3c979cbb14085b2d858f4f0785f26: acquiring CA information for follower catalog manager: success
I20260927 04:02:49.251605 6485 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8bd3c979cbb14085b2d858f4f0785f26: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 04:02:49.446542 6639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:49.443810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32882 (local address 127.15.185.125:33025)
0927 04:02:49.443974 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:49.443976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:49.443986 (+ 10us) server_negotiation.cc:407] Connection header received
0927 04:02:49.444013 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:49.444014 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:49.444025 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:49.444138 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:02:49.444261 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:49.444901 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:49.445759 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:49.446016 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:49.446082 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:49.446228 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:02:49.446257 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:02:49.446300 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:49.446408 (+ 108us) server_negotiation.cc:299] Negotiation successful
0927 04:02:49.446428 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":82,"threads_started":1}
I20260927 04:02:49.450902 6645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:49.448371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35064 (local address 127.15.185.124:41687)
0927 04:02:49.448508 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:49.448510 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:49.448518 (+ 8us) server_negotiation.cc:407] Connection header received
0927 04:02:49.448566 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:49.448567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:49.448622 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:49.448691 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:02:49.448838 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:49.449309 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:49.450016 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:49.450229 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:49.450285 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:49.450471 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:02:49.450580 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:02:49.450711 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:49.450746 (+ 35us) server_negotiation.cc:299] Negotiation successful
0927 04:02:49.450764 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":76,"threads_started":1}
I20260927 04:02:49.455591 6639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:49.452594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32888 (local address 127.15.185.125:33025)
0927 04:02:49.452707 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:49.452708 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:49.452903 (+ 195us) server_negotiation.cc:407] Connection header received
0927 04:02:49.453003 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:49.453004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:49.453015 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:49.453048 (+ 33us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:02:49.453212 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:49.454013 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:49.454841 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:49.455113 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:49.455181 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:49.455327 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:02:49.455362 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:02:49.455414 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:49.455481 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 04:02:49.455508 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 04:02:49.455972 6645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:49.452781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35072 (local address 127.15.185.124:41687)
0927 04:02:49.452842 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:49.452844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:49.452924 (+ 80us) server_negotiation.cc:407] Connection header received
0927 04:02:49.452970 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:49.452971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:49.452983 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:49.453038 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:02:49.453439 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:49.454147 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:49.455134 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:49.455440 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:49.455497 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:49.455670 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:02:49.455703 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:02:49.455756 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:49.455892 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 04:02:49.455921 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 04:02:49.456375 6652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:49.452605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60442 (local address 127.15.185.126:38265)
0927 04:02:49.452830 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:49.452832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:49.452950 (+ 118us) server_negotiation.cc:407] Connection header received
0927 04:02:49.453108 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:49.453110 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:49.453169 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:49.453268 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:02:49.453394 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:49.453977 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:49.455118 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:49.455460 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:49.455519 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:49.455871 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:02:49.455996 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:02:49.456054 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:49.456254 (+ 200us) server_negotiation.cc:299] Negotiation successful
0927 04:02:49.456290 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":48,"threads_started":1}
I20260927 04:02:49.458366 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 6235
I20260927 04:02:49.463538 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 6310
I20260927 04:02:49.469209 16101 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTmmW3q/build/release/bin/kudu with pid 6385
2026-09-27T04:02:49Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2903 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5755 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (271459 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST